Google is Broken - And I think I know how...

[Joeychgo]

By: Joeychgo
Please DIGG this article

Over the past weeks, many website owners and forum owners have been complaining that Google is deindexing their sites, moving the results to supplemental results, and that they are losing position in the SE Rankings.

One of the search terms I watch is for the phrase vBulletin. Mainly because I own a site that depends on the search term vBulletin. (www.vbulletin-faq.com)

Its a good search term to watch because who wants to spam for that keyword? Nobody. So, its a good way to see how relevant the rankings are, and to a certain extent, gives you clues about what Google uses to determine rankings.

But you cant really do that now. Why? Google's SE rankings are completely irrelevant at this time.

How do I know this? This is a subjective determination usually. But, I think its clear.

Lets look.

Google's Top 10 Rankings

1. www.vbulletin.com/
2. en.wikipedia.org/wiki/VBulletin
3. www.packtpub.com/vbulletin/book
4. www.securityfocus.com/archive/1/437817
5. www.securityfocus.com/bid/16128
6. www.tivocommunity.com/tivo-vb/
7. akismet.com/blog/2006/06/for-vbulletin/
8. www.hardforum.com/
9. forum.time2dine.co.nz/seo-vbulletin/ search-engine-optimize-vbulletin-98.html
10. www.nvnews.net/vbulletin/

1,3 and 9 belong as far as I see it. Those are logical results. Even wiki (2) could be considered relevant, although its my personal opinion that wiki is worthless because of its senseless and continous change and spamming.


4 & 5 are talking about a VBulletin Profile.PHP Cross-Site Scripting Vulnerability but contain virtually no information. But its a PR8 Site.


6, 8 and 10 are just forums that run on vBulletin. They have nothing to do with vBulletin. (i.e. Tivo)

Lets look at the next 10.

1. www.nvnews.net/vbulletin/
2. www.nvnews.net/vbulletin/forumdisplay.php?f=14
3. tt.tennis-warehouse.com/
4. forums.winamp.com/
5. forums.anarchy-online.com/
6. www.vbadvanced.com/
7. www.mp3car.com/vbulletin/
8. www.mambers.com/
9. fileforums.com
10. www.urban75.net/vbulletin/

6 is relevant. MAYBE 8 is as well. All the rest are just forums about various topics, and using vBulletin software.


So out of the top 20, we have 5 sites that belong there. Thats it.


Lets look at Yahoo...

1. www.vbulletin.com
2. www.vbulletin.org
3. www.vbulletin-faq.com
4. www.vbmechanic.com
5. www.vbulletin.org/forum
6. www.vbstyles.com
7. en.wikipedia.org/wiki/VBulletin
8. www.vbulletin-fans.com
9. www.vbulletin-faq.com/forum
10. www.jelsoft.com

EVERY site is directely related to vBulletin and what a reader would expect to find when searching for that term.



How about MSN (they only list 8 on a page)

1. www.vbulletin.com
2. www.vbulletin.com/forum
3. www.vbulletin.org/forum/portal.php
4. www.vbwebmaster.com/forums
5. www.vbulletin-faq.com/forum
6. www.vbseo.com/vbulletin
7. www.siteground.com/vbulletin-hosting.htm
8. en.wikipedia.org/wiki/VBulletin

Again, all are relevant.


How about ASK.com

1. www.vbulletin.com/
2. www.extremepixels.com/
3. www.vbstyles.com/
4. vbadvanced.com/
5. www.vbulletin-fr.org/
6. www.best-board.com/
7. www.cinvin.com/
8. www.vbulletin.org/
9. www.vbwebmaster.com/
10. www.transversestyles.com/

Once again - all relevant results.


Ok - so now we know the Google results are not relevant, at least not as relevant as the other major search engines.

Why?

This, nobody can say with certainty. But, I think I have an idea...

Since Big Daddy, many many sites have been deindexed, in whole or part, or their pages have been moved to Google's "Supplemental results" index.


Big Daddy, was a Google infrastructure update that was completed about 2 months ago. basically, it was a major update to how they index and crawl, and subsequently list sites in their index. It was suppose to be more helpful in detecting spam sites.

But Matt Cutts, Google Engineer, recently acknowleged that Google doesnt always think of smap the same as a user:

I heard that recently from a Googler, too. Sometimes we think of spam as strictly things like hidden text, cloaking, etc. But users think of spam as noise: things that they don’t want. If they’re trying to get information, fix a problem, read reviews, etc., then sites that like aren’t as helpful.

Ok, so we had this big change at Google. Then sites were insanely indexed. One of my sites had 400,000 pages indexed which is abot double what it ever had before.

Then the bottom fell out. Pages started moving to supplemental, and being dropped completely.

This wasnt just me. This has happened on many many many sites.

Then... People started losing their rankings. Sites that were previously in the top ten for years had dropped, often considerably. Many also fell completely out of the index for terms they had ranked highly for.

Why?

Simple. One of the main factors Google uses for determining rankings is how many sites link into your site, and what anchor text is used in those links as well as the subject of the text surrounding the link.

When a page is either deindexed or moved to supplemental, any outgoing links on that page are ignored. So, while Google is busy de-indexing pages from forums, blogs, directories, etc, they are deindexing the links that point to the relevant sites.

Forums in particular are a great source of organic links. Why do I say this? Because forum members pass links around to each other to relevant sites. Its the PEOPLE that use the internet making the links as a trade of information. When forums and other similar sites are deindexed, the outgoing links are lost as well.

Whats often left, are the major spammers who have vast interlinked networks, or sites that are highly ranked, but jsut have a mention of the topic and arent really relevant to that topic. A good example are how many forums appear in Google's top 20 listing. vBulletin is mentioned in the title (powered by vBulletin) and in the footer. Thats it. They aren't discussiing vBulletin on TIVO's forum!

Conclusion...

From the above examples, we can see Google isn't as relevant as it could be or should be. THis will continue until Google finds a way to fix their indexing issues.

My recommendations...

Dont change your SEO one bit for Google. Direct your SEO efforts toward Yahoo, MSN and ASK. Ignore Google, at least until they fix whats broken.

Sidenote...

I believe this is a great chance for Yahoo and Ask to knock Google off its pedestal. If I owned Google stock, I would be selling it right now.



Please Digg This Article

[Brandon Sheley]

some solid examples u got

[minstrel]

I agree with most of what you have said, Joey. People have been saying, erroneously, that "Google is broken" for years. As I have said elsewhere, for the first time I think all the evidence is telling us that since Big Daddy it really is broken this time.

I'll give you my own example. Several months ago, I converted my Psychlinks forum from phpBB to SFM. I did all the usual appropriate redirects and deleted the old phpBB files to elininate the duplicate factor issue. At the time, I had several thousand forum pages indexed (I've forgotten exactly how many but it was well in execess of 20K). Google rather quickly reindexed my new forum - not 20K but several thousand and the phpBB pages were disappearing quickly.

Thenn we come to Big Daddy. I watched my SMF pages dwindle rapidly, down to just the index page at its worst. And at the same time, all those old phpBB pages started appearing again. At one point, looking at the entire psychlinks.ca, Google was reporting about 8500 pages and MOST of those were pages that had beend deleted six months earlier.

Today, I am slowly seeing the number of my forum pages returning (up to a whopping 82 today) and bit by bit the phpBB pages are dropping out again. But the question is: Why did they reappear at all? They didn't exist!

The rhetoric from Matt Cutts in his now infamous "timeline" blog was a double slap in the face. There he claimed that the people "damaged" by Big Daddy were people who were using Black Hat SEO techniques or artifical link schemes. That is, to be blunt, absolute bullshit.

I have NEVER, repeat NEVER used anything even remotely Black Hat or even Gray hat on my sites. I'm in this for the long term and I have never believed in risking the long-term for short term gain.

The only time I have ever been involved in anything remotely like a link scheme was the DigitalPoint Coop for a few months. But I became disillusioned with and uneasy about that in October or early November 2005 and dumped it, LONG before Big Daddy. I was using it on the old phpBB forum but never on the SMF version.

At the same time as the old nonexistent phpBB pages began reappearing in the index, I noticed other pages (static HTML pages from my main site) also reappearing that had disappeared in a site redesign back around June or July 2005. Indeed, the cache dates on these additiona nonexistent pages confirmed that because they were for the most part dated July 2005.

My conclusion?

Something went terribly wrong with Big Daddy and it was widespread. In attempting to repair the damage, Google was and still is attempting to repair the index from old backups from as long ago as mid-2005. In the meantime, there search results are utter crap. I think part of this has to do with their new crawling and caching routines, which simply are not working very well at all.

We could all probably live with this, for a while, IF Google had simply come clean. What is most aggravating to me is the long and ongoing litany of bafflegab and bullshit - alternating with dead silence - coming from Google lately on the issues with their search engine.

Forget about Googlemaps, Google Sitemaps. Google Chat, and Gbuy, Google. Get back to what got you where you were.

Fix the damn search engine. Or you become the next dinosaur.

[KW802]

Joey,

Good article; very concise & straight to the point of the importance of relevancy versus popularity. Also quite a pleasant surprise to see one of my sites listed in Ask's top 10 list.

Kevin

[Joeychgo]

Joey,

Good article; very concise & straight to the point of the importance of relevancy versus popularity. Also quite a pleasant surprise to see one of my sites listed in Ask's top 10 list.

Kevin

Hee hee - I thought you would like that..


I think Big Daddy was intended to do one thing - but that one thing has impacted others, and then created a domino effect.

As you know PR has to do with incoming links. PR has some play in how often and how deeply your site is indexed. As sites are deindexed, the outgoing links are not counted. When a link to you from there is lost, your PR decreases and your ranking for that KW declines. Now your outgoing links dont count either. Etc etc.

[Noppid]

I too pulled the coop. I was running site wides for nothing since the updates.

Until the coop diversifies and uses multiple search engines for balance, and the fact google is broken changes, the coop is old news IMO. But I hope the adjustments are made.

I mean the google api was for the old search engine and in beta. How did we get a new search engine and not a new api? That makes no sense to me.

[minstrel]

How did we get a new search engine and not a new api? That makes no sense to me.

Perhaps for the same reason we got a broken link: query and a broken site: query.

[Noppid]

My reference for my assumption is that amazon has 3 versions of their API currently available. But I'm no expert at API implementation.

[EarlGrey]

Superb analysis and presentation.
Impressive post.

[mcfox]

Good article, Joey.

(about time I posted here )

Dugg, of course.

[Peggy]

yes fox, it IS about time

[dazzlindonna]

Nice presentation of facts and theories. I've blogged it on seo-scoop. Would love to see more.

[Joeychgo]

Nice presentation of facts and theories. I've blogged it on seo-scoop. Would love to see more.

Thank You

[Elpie]

Hmmm... well, that explains a lot! Thanks for the post. I have been out of the SEO loop for the past couple of months and was unaware of the changes with Google, while tearing my hair out over not being able to get a site listed and wondering what had happened to another site of mine.

Three months ago a site I own that is the last of the sites still actively assisting people to be reunited following last years hurricanes had all its pages ranked. The index had a page rank of 8 and we had thousands of relevant incoming links. Google had reported increased traffic from them steadily in the period until May and with an average 2 million page views a month this year, we were doing okay. Last month, almost all pages were dropped from Google and the only page with any ranking, the index, is now at 3!
The Spanish language site we run with it is not listed at all. On a search for the incoming links that are listed, all links from sites such as CNN, MSNBC, and other news sites have disappeared.
Well, we all know that last years hurricanes are old news, but this delisting is just ridiculous!
To make it even more ridiculous, Google returns many of these same news stories if I search on my name. Site name not indexed, but my name is - go figure

[Peggy]

an excellent, worthwhile cause. I hope the site gets re-indexed quickly

[BamaStangGuy]

This is the obsulute greatest example of how screwed up Google is. It is about time someone stepped up and proved it. Good job Joey!

[Noppid]

This will involve the SEC soon IMO. It's obvious that the organic part is non-existant and that the point is to send the adsense and adwords money in a circle through google. Maybe it's legal, but it's certainly useless to web surfers and at least unethical IMO if that is the case.

Do no evil? Yeah, right!

[Joeychgo]

Well, Kinderstart has brought suit. It will be interesting to see how the legal implications are worked out on that one.

[Big Dan]

Personally, I'm sick of Google! They've become the defacto standard when doing SEO, it seems as if nobody cares about the other big engines. Putting all your eggs in one basket is not a good idea. Google screwed up and now you're paying the price.

[Joeychgo]

I agree completely