Monthly Archives: May 2016

WordPress.org Hosting Recommendations Listing Criteria

UPDATE (5/13/2016 7:05 PM): Official comment from Matt Mullenweg was posted. Quoted below, click or scroll to the comment section to see the original.

“I would like to see some transparency in the process”

As stated on the page, the listing is completely arbitrary. The process was: There was a survey, four applicants were chosen, and the page was updated. That might repeat later in the year, or the process might change.

“how criteria are weighted”

There is no criteria or weighting. It ultimately is one person’s opinion. Information that is important is reflected in the questions asked in the survey, but that is not everything that is taken into account. (I have looked at this site in the past, for example.)

“who is making the decisions”

I am. James helped in sorting through the many submissions that came in, doing additional research, and digging on finalists, but ultimately the decision was mine. You can and should blame me fully for any issues you have with it. I appreciate James’ help in this go-round, but he will not be involved at all with any future updates. (So, please leave him alone.)

“how much money is involved”

There was no money involved. Obviously being listed on the page is hugely valuable and impacts the listed (or unlisted) businesses a great deal. This is why I take full responsibility for the listing, now and in the future — I have been fortunate to be extraordinarily successful and no financial or business consideration any of the applicants could offer matters to me. A host could offer $100,000,000 to be listed on the page for 1 day, and I would say no.

-Matt Mullenweg


Yesterday, I posted WordPress.org Updates Hosting Recommendations, Nobody Knows Selection Criteria. Which naturally meant I was going to find out as much as I could about the process, because it's a big deal and my mission here at Review Signal is honest and transparent web hosting reviews.

I confirmed with multiple sources that the newly listed companies didn't pay any money to get listed. Everyone seems to have filled out the form and then heard nothing back until the updated page was published yesterday. Both the winners (BlueHost [Reviews], DreamHost [Reviews], FlyWheel [Reviews], SiteGround [Reviews]) and losers (everyone else) seemed to agree on this process based on everyone I talked to.

Great. The application process seems fair.

But the selection process is still a black box, with help from people who follow WordPress more closely than myself, I found James Huff (macmanx) a 12 year volunteer and 5 year employee at Automattic who was directly involved with the new WP.org hosting recommendations.

James_huff1

I didn't hide who I was or my interest. The most concerning part of this exchange was that 'Absolutely no money changed hands, unless you consider sponsorship of WordCamps as monetary with regards to the "contributions to WordPress.org."'

No money changed hands except a lot of sponsorship dollars to the organization. Guess who the top global gold community sponsors are? BlueHost (and JetPack/WooCommerce, both owned by Automattic). Somehow BlueHost are also a Silver sponsor too, along with GoDaddy. BlueHost is pouring a lot of money into WordCamps/WordPress.org Foundation.

I'm sorry, but I do consider that money changing hands. They are giving a large sum of money - it's material enough to mention in their SEC filings.

James_huff2

We're still going to have to agree to disagree about what money changing hands means. But he says it was fair. But fair is pretty meaningless when we don't really have any insight into what standard of fairness is the goal. How is each criteria being weighed and evaluated. But this is the list of hosts that they can confidently tell everyone are good.

I'm not sold.

James_huff3

Historical perception seems to be the proxy for what marketers might call Net Promoter Score (NPS). How much do consumers like/recommend something. That's essentially what I measure here at Review Signal and my data has been incredibly close to what company's internal data shows (LiquidWeb NPS Scores vs LiquidWeb Review Signal Rating).

It is arguably the most important factor of recommendations and for service businesses, it's about the best metric for all encompassing quality available.

But it's only part of the criteria and that's fair. But should there be some minimum threshold? Can a company score a zero in quality and high in everything else be worthy of a listing? BlueHost's rating is 41%. That means roughly 6/10 people don't recommend it or have anything good to say about them.

There are WordCamp sponsors that didn't make the cut. Of the global community sponsors 2/3 hosting companies did though, BlueHost and DreamHost, while one didn't, GoDaddy. But the largest sponsor made it and is at the top and it's still BlueHost.

But moving on, James mentioned Automattic has no play in the process, but he does wear multiple hats. Which means he is aware of the potential perception of conflict of interest.

James_huff4

Finally, a mention of Matt. Important again when thinking about the context for potential conflicts of interest. I outline what would happen in a dream world and what's realistic. I think honest disclosure and basic transparency is perfectly realistic. It's ok to make money, just be clear about where it's coming from. A standard I try to uphold here at Review Signal, see how we make money and read the entire process for how our rankings are calculated. See? It's not hard and I still make money giving the best information available.

James_huff5

AWP comments

That is the comment thread I referenced. Not a single person said anything positive about BlueHost and the assumption is they just paid for it. BlueHost being listed ruins the credibility of the recommendations when there is no transparency about what criteria was being used.

James_huff6

Moving on, the survey itself has issues which I brought up before. It's asking for sensitive company information and being handled by employees of a company that owns two competitors in the space (WP.com VIP, Pressable), took $15 million in investment from another (BlueHost), and is an investor in a fourth competitor (WP Engine).

That seems like a huge potential conflict of interest and I know it dissuaded at least one company from even applying.

James_huff7

james huff 3 tweets

It didn't end on the nicest note, I don't think James took my criticisms well. From his original messages, I think he knows and understands the perception of conflicts of interest but admitting them in this context puts him in a very awkward position that I don't envy. He wears multiple hats and surely wants to wear them all fairly. I would say admitting that those multiple hats has the potential for conflicts of interest isn't a weakness of character, it's an admission of humanity. I'm sure James is a great guy and has done a lot of good things for the community. But I think people who can be perceived with a strong potential for conflict of interest, which anyone connected to Automattic in this situation would have, shouldn't be managing this particular process.

I truly don't have any ill will towards James personally or Automattic. Even BlueHost/EIG, I've been more than willing to give them the benefit of the doubt and continue to hope that they will be better (ASO did break my heart a bit, I thought they were turning EIG around). My data continues to show them being mediocre and a seeming touch of death in terms of quality (their strategy does seem to be cost cutting and economies of scale). But I don't fault them for their behavior, I expect it, it's well published in their SEC filings.

Conclusion

I still think WordPress.org can do better with its hosting recommendations and I'm not going to stop advocating for them until they are better. I would like to see some transparency in the process, how criteria are weighted, who is making the decisions and how much money is involved. I think the companies that applied would appreciate feedback about why they weren't selected, what makes them different and fall short of the companies that do make the cut. Or just call them Ads / Sponsors. Don't say they are the best and brightest and endorse them. Say, we took money and this guy paid us the most. At least we meet the minimum threshold of honesty and transparency.

 

References

For posterity, the logs in their entirety are available below. It's long, so I tried to cut down some stuff to get to the most important bits. But I don't want to hide anything.

Direct Message Archive macmanx Making WordPress Slack Direct Message Archive macmanx Making WordPress Slack2

WordPress.org Updates Hosting Recommendations, Nobody Knows Selection Criteria

I've railed about Drupal and WordPress Have Sold Us Out in terms of hosting recommendations before. We've been waiting a long time (around a year now?) for WordPress.org to do it's revamp of its hosting recommendation page.

The Winners

BlueHost, DreamHost, FlyWheel, SiteGround

I'm not shocked at all to see BlueHost somehow still manages to be at the very top (albeit the list is alphabetical). They've continuously survived being listed, I guess that's what a million dollars will do.

Where is the transparency?

They requested hosts submit a ridiculous amount of personal information. You can see the full survey below:

2016 WordPress Hosting Survey - WordPressorg

It asks some deeply private questions like number of employees, how many 30 day active paying customers you have, and how many net paying customers are you gaining or losing each month?

Mind you, as far as anyone can tell, Matt has complete control over who shows up, and Automattic bought the majority stake in a company competing in the WordPress hosting space, Pressable. They also run WordPress.com VIP. They are also an investor in WPEngine. So some of the most secretive numbers a company competing in this space might have are being disclosed potentially to multiple of their biggest competitors through a process with no transparency or even a person named to be responsible for it.

That alone is worrisome for the process, it should definitely be run independent of Matt.

Everything else needs to be explained too. Who is responsible for this revamp? What were the selection criteria? How often will it be updated? Will existing companies be continuously re-evaluated?

wordpress_org_listing

It's not clear who 'we' is. They say listing is arbitrary but then add criteria. I'm not sure they understanding what arbitrary means. Or maybe they simply ignore the criteria they mention. Maybe it's just a terrible joke? Just like the process (or lack thereof) that seems to be in place.

A lot of it is pretty subjective. design, tone, ease of WP auto-install, historical perception? BlueHost is still listed, which is has consistently been pretty poorly reviewed (along with just about all EIG brands) and continues a downward trend.

BlueHost_review_signal_rating_apr_2016

Furthermore, it's the same criteria that's been written since at least 2010.

So maybe saying it's arbitrary gives them as escape to list whomever they want, especially considering the financial considerations involved.

Newly Listed Companies

I tried to find some explanation for how the three new companies were selected, but there really isn't much to go on. DreamHost is a Silver Community Sponsor for WordCamp, but so is GoDaddy who did not make the cut.

FlyWheel only does WordPress, but DreamHost and SiteGround do a lot more.

DreamHost has a ton of forum threads on WordPress.org, SiteGround has only a few over 10 years. FlyWheel has one total.

I talked to someone at one of the newly listed hosted companies and they confirmed that the form was filled out and that was it. Also, there was no financial consideration involved with the listing.

Which is very nice to hear, but doesn't really inspire confidence in the recommendations.

I've aired my concern with BlueHost multiple times.

But what about the new companies and their ratings?

DreamHost has a 59% rating on Review Signal, which is ok, given the upper end of the shared hosting spectrum is SiteGround at 71%. FlyWheel, the specialized hosting company has the highest rating of any company at a whopping 85%.

So the new companies are all far better than BlueHost (41%). But there are other very highly rated companies that didn't make the cut. For example, WP Engine (72%) is probably the biggest  name not listed based on size, brand in the WP community and rating at Review Signal.

Conclusion

I'm glad there are some much better companies than Blue Host listed and at least one of them got there without paying for the privilege. There is still language about some donating a portion of the fee back, which makes you think it's still at least BlueHost.

I'm still unhappy with the lack of transparency of the entire process. The most influential place for people entering the WordPress community is recommending one very mediocre hosting company who has historically paid large sums to be listed and has a deep financial relationship with the person ultimately responsible for the recommendations. The revamp didn't change that.

I am disappointed and I don't expect to hear anything from WordPress.org/Matt clarifying the hosting page, again.

 

UPDATES

(5/13/2016)

There was a little discussion in the WordPress slack. macmanx is James Huff, an Automattic employee. Seems they wanted only 1 managed WordPress host. Other details include around 100 applications. And even in the WordPress slack, the first comment doubts that these are really the best (well, one which almost everyone assumes to be BlueHost).

james_huff_3_outta_4 James_Huff_hosting_recommendations

Loading...

Interested in seeing which web hosting companies people love (and hate!)? Click here and find out how your web host stacks up.