Google faces its next big challenge

There is, in the words of Jeff Atwood, "trouble in the house of Google". It's not unrest within the company that he's talking about, though; it's externally among users who are beginning to find that when they try to do searches to evaluate or buy consumer items - such as dishwashers, or iPhone 4 cases - or to find a site that will give them some useful answers, that Google's results are awash with spam.

In fact, the problem that plagued the first generation of search engines such as Altavista now seems to be gaining traction on Google, which outdistanced those earlier rivals precisely because it dumped the spam so effectively.

Paul Kedrosky ("investor, speaker, writer, media guy, and entrepreneur") noted in a plaintive post in mid-December that

    "Over the weekend I tried to buy a new dishwasher. Being the fine net-friendly fellow that I am, I  began Google-ing for information. And Google-ing. and Google-ing. As I tweeted frustratedly at the tend of the failed exercise, 'To a first approximation, the entire web is spam when it comes to appliance reviews'."

Kedrosky noted that "Google has become a snake that too readily consumes its own keyword tail. Identify some words that show up in profitable searches - from appliances, to mesothelioma suits, to kayak lessons - churn out content cheaply and regularly, and you're done. On the web, no-one knows you're a content-grinder."

He also adds that "Google has to know this. The problem is too big and obvious to miss."

And indeed Google does know about this. Jeff Atwood at Coding Horror - one half of the creators of Stack Overflow (the other half being Joel Spolsky) and the other "Overflow" sites (which let people ask questions on a topic and the better ones get voted up - like a, ahem, better version of Quora) - had already noticed how sites which scrape Stack Overflow and its siblings actually rank better on Google than the original.

Now, Stack Overflow allows scraping, as long as there's a link back to the original (which cannot be a nofollow); yet even with this, it ranked below the scraping sites.

Atwood consulted Matt Cutts, Google anti-spam king:

    "We did a ton of due diligence on webmasters.stackexchange.com to ensure we weren't doing anything overtly stupid, and uber-mensch Matt Cutts went out of his way to investigate the hand-vetted search examples contributed in response to my tweet asking for search terms where the scrapers dominated. Issues were found on both sides, and changes were made. Success!"

Except it isn't really success. It's a temporary respite. As Atwood points out moments later, "Anecdotally, my personal search results have also been noticeably worse lately. As part of Christmas shopping for my wife, I searched for 'iPhone 4 case' in Google. I had to give up completely on the first two pages of search results as utterly useless, and searched Amazon instead."