WordPress.org Hosting Recommendations Listing Criteria

UPDATE (5/13/2016 7:05 PM): Official comment from Matt Mullenweg was posted. Quoted below, click or scroll to the comment section to see the original.

“I would like to see some transparency in the process”

As stated on the page, the listing is completely arbitrary. The process was: There was a survey, four applicants were chosen, and the page was updated. That might repeat later in the year, or the process might change.

“how criteria are weighted”

There is no criteria or weighting. It ultimately is one person’s opinion. Information that is important is reflected in the questions asked in the survey, but that is not everything that is taken into account. (I have looked at this site in the past, for example.)

“who is making the decisions”

I am. James helped in sorting through the many submissions that came in, doing additional research, and digging on finalists, but ultimately the decision was mine. You can and should blame me fully for any issues you have with it. I appreciate James’ help in this go-round, but he will not be involved at all with any future updates. (So, please leave him alone.)

“how much money is involved”

There was no money involved. Obviously being listed on the page is hugely valuable and impacts the listed (or unlisted) businesses a great deal. This is why I take full responsibility for the listing, now and in the future — I have been fortunate to be extraordinarily successful and no financial or business consideration any of the applicants could offer matters to me. A host could offer $100,000,000 to be listed on the page for 1 day, and I would say no.

-Matt Mullenweg


Yesterday, I posted WordPress.org Updates Hosting Recommendations, Nobody Knows Selection Criteria. Which naturally meant I was going to find out as much as I could about the process, because it's a big deal and my mission here at Review Signal is honest and transparent web hosting reviews.

I confirmed with multiple sources that the newly listed companies didn't pay any money to get listed. Everyone seems to have filled out the form and then heard nothing back until the updated page was published yesterday. Both the winners (BlueHost [Reviews], DreamHost [Reviews], FlyWheel [Reviews], SiteGround [Reviews]) and losers (everyone else) seemed to agree on this process based on everyone I talked to.

Great. The application process seems fair.

But the selection process is still a black box, with help from people who follow WordPress more closely than myself, I found James Huff (macmanx) a 12 year volunteer and 5 year employee at Automattic who was directly involved with the new WP.org hosting recommendations.

James_huff1

I didn't hide who I was or my interest. The most concerning part of this exchange was that 'Absolutely no money changed hands, unless you consider sponsorship of WordCamps as monetary with regards to the "contributions to WordPress.org."'

No money changed hands except a lot of sponsorship dollars to the organization. Guess who the top global gold community sponsors are? BlueHost (and JetPack/WooCommerce, both owned by Automattic). Somehow BlueHost are also a Silver sponsor too, along with GoDaddy. BlueHost is pouring a lot of money into WordCamps/WordPress.org Foundation.

I'm sorry, but I do consider that money changing hands. They are giving a large sum of money - it's material enough to mention in their SEC filings.

James_huff2

We're still going to have to agree to disagree about what money changing hands means. But he says it was fair. But fair is pretty meaningless when we don't really have any insight into what standard of fairness is the goal. How is each criteria being weighed and evaluated. But this is the list of hosts that they can confidently tell everyone are good.

I'm not sold.

James_huff3

Historical perception seems to be the proxy for what marketers might call Net Promoter Score (NPS). How much do consumers like/recommend something. That's essentially what I measure here at Review Signal and my data has been incredibly close to what company's internal data shows (LiquidWeb NPS Scores vs LiquidWeb Review Signal Rating).

It is arguably the most important factor of recommendations and for service businesses, it's about the best metric for all encompassing quality available.

But it's only part of the criteria and that's fair. But should there be some minimum threshold? Can a company score a zero in quality and high in everything else be worthy of a listing? BlueHost's rating is 41%. That means roughly 6/10 people don't recommend it or have anything good to say about them.

There are WordCamp sponsors that didn't make the cut. Of the global community sponsors 2/3 hosting companies did though, BlueHost and DreamHost, while one didn't, GoDaddy. But the largest sponsor made it and is at the top and it's still BlueHost.

But moving on, James mentioned Automattic has no play in the process, but he does wear multiple hats. Which means he is aware of the potential perception of conflict of interest.

James_huff4

Finally, a mention of Matt. Important again when thinking about the context for potential conflicts of interest. I outline what would happen in a dream world and what's realistic. I think honest disclosure and basic transparency is perfectly realistic. It's ok to make money, just be clear about where it's coming from. A standard I try to uphold here at Review Signal, see how we make money and read the entire process for how our rankings are calculated. See? It's not hard and I still make money giving the best information available.

James_huff5

AWP comments

That is the comment thread I referenced. Not a single person said anything positive about BlueHost and the assumption is they just paid for it. BlueHost being listed ruins the credibility of the recommendations when there is no transparency about what criteria was being used.

James_huff6

Moving on, the survey itself has issues which I brought up before. It's asking for sensitive company information and being handled by employees of a company that owns two competitors in the space (WP.com VIP, Pressable), took $15 million in investment from another (BlueHost), and is an investor in a fourth competitor (WP Engine).

That seems like a huge potential conflict of interest and I know it dissuaded at least one company from even applying.

James_huff7

james huff 3 tweets

It didn't end on the nicest note, I don't think James took my criticisms well. From his original messages, I think he knows and understands the perception of conflicts of interest but admitting them in this context puts him in a very awkward position that I don't envy. He wears multiple hats and surely wants to wear them all fairly. I would say admitting that those multiple hats has the potential for conflicts of interest isn't a weakness of character, it's an admission of humanity. I'm sure James is a great guy and has done a lot of good things for the community. But I think people who can be perceived with a strong potential for conflict of interest, which anyone connected to Automattic in this situation would have, shouldn't be managing this particular process.

I truly don't have any ill will towards James personally or Automattic. Even BlueHost/EIG, I've been more than willing to give them the benefit of the doubt and continue to hope that they will be better (ASO did break my heart a bit, I thought they were turning EIG around). My data continues to show them being mediocre and a seeming touch of death in terms of quality (their strategy does seem to be cost cutting and economies of scale). But I don't fault them for their behavior, I expect it, it's well published in their SEC filings.

Conclusion

I still think WordPress.org can do better with its hosting recommendations and I'm not going to stop advocating for them until they are better. I would like to see some transparency in the process, how criteria are weighted, who is making the decisions and how much money is involved. I think the companies that applied would appreciate feedback about why they weren't selected, what makes them different and fall short of the companies that do make the cut. Or just call them Ads / Sponsors. Don't say they are the best and brightest and endorse them. Say, we took money and this guy paid us the most. At least we meet the minimum threshold of honesty and transparency.

 

References

For posterity, the logs in their entirety are available below. It's long, so I tried to cut down some stuff to get to the most important bits. But I don't want to hide anything.

Direct Message Archive macmanx Making WordPress Slack Direct Message Archive macmanx Making WordPress Slack2

The following two tabs change content below.
avatar
Kevin Ohashi is the geek-in-charge at Review Signal. He is passionate about making data meaningful for consumers. Kevin is based in Washington, DC.





9 thoughts on “WordPress.org Hosting Recommendations Listing Criteria

  1. avatarMatt

    “I would like to see some transparency in the process”

    As stated on the page, the listing is completely arbitrary. The process was: There was a survey, four applicants were chosen, and the page was updated. That might repeat later in the year, or the process might change.

    “how criteria are weighted”

    There is no criteria or weighting. It ultimately is one person’s opinion. Information that is important is reflected in the questions asked in the survey, but that is not everything that is taken into account. (I have looked at this site in the past, for example.)

    “who is making the decisions”

    I am. James helped in sorting through the many submissions that came in, doing additional research, and digging on finalists, but ultimately the decision was mine. You can and should blame me fully for any issues you have with it. I appreciate James’ help in this go-round, but he will not be involved at all with any future updates. (So, please leave him alone.)

    “how much money is involved”

    There was no money involved. Obviously being listed on the page is hugely valuable and impacts the listed (or unlisted) businesses a great deal. This is why I take full responsibility for the listing, now and in the future — I have been fortunate to be extraordinarily successful and no financial or business consideration any of the applicants could offer matters to me. A host could offer $100,000,000 to be listed on the page for 1 day, and I would say no.

    Reply
    1. avatarKevin Ohashi Post author

      I wrote this to you privately, but I want to put it here as well.

      First, thank you for commenting.

      I respect that you’re taking full ownership, I hope you understand when I still have disagreements with the process, calling it arbitrary and having criteria are at odds. My door is always open to you about how I think it could be done and/or handled better and more transparently. I think it’s such an important page for such an important community, that it needs to be better.

      Reply
  2. avatarSteven Gliebe

    Kevin, I sold HostingReviews.io and the new owner has it mostly up to date now. BlueHost is still in the bottom 30% according to what customers are saying on Twitter. No improvement. Of those that rank lower, half of them are EIG brands. I’ve seen basically same thing as you for as long as I collected data.

    The data shows that there are better options. Two of them were added beneath Bluehost on that page but I feel strongly that continuing to recommend Bluehost is an unnecessary disservice to WordPress users.

    Reply
  3. avatarChristopher Neetz

    Excuse me, why can’t this page be a searchable list, with tags and backlinks? Shouldn’t we be rating and commenting on hosting companies openly? Shouldn’t the community give feedback on what is working for them and WordPress? This page is a joke and either needs to be removed or the idea revised completely.

    Reply
  4. avatarNemanja Aleksic

    The recommendation page has been updated. Bluehost and SiteGround are gone, apparently after the core team found out that BH and SG seem to be messing with wp-config.php on install and imposing some kind of limitations and force plugins updates.

    Reply
      1. avatarNemanja Aleksic

        Hm, interesting. There was a thread by Paul Dahlen on the Advanced WordPress Facebook group about Bluehost and SiteGround being taken down, and one of the commenters stated the reason for the takedown. It went into more details, but what I posted here was the jist of it.

        https://www.facebook.com/groups/advancedwp/
        I checked the thread to reference the source, but the comment has since been deleted. Could have been someone talking out of their ***, but so far it’s the only explanation put forward by anyone.

        The plot thickens…

        Reply
  5. Pingback: The WordPress.org Recommended Hosting Page is Revamped, Features Flywheel for the First Time – WordPress Tavern

Leave a Reply to Nemanja Aleksic Cancel reply

Your email address will not be published. Required fields are marked *

Current day month ye@r *

Loading...

Interested in seeing which web hosting companies people love (and hate!)? Click here and find out how your web host stacks up.