Joeychgo
03-29-2006, 08:48 AM
Matt Cutts, a Google Engineer who posts a blog to help webmasters, recently haled an open Q & A on his blog.
One question and answer caught my attention...
Q: “My sitemap has about 1350 urls in it. . . . . its been around for 2+ years, but I cannot seem to get all the pages indexed. Am I missing something here?”
A: One of the classic crawling strategies that Google has used is the amount of PageRank on your pages. So just because your site has been around for a couple years (or that you submit a sitemap), that doesn’t mean that we’ll automatically crawl every page on your site. In general, getting good quality links would probably help us know to crawl your site more deeply. You might also want to look at the remaining unindexed urls; do they have a ton of parameters (we typically prefer urls with 1-2 parameters)? Is there a robots.txt? Is it possible to reach the unindexed urls easily by following static text links (no Flash, JavaScript, AJAX, cookies, frames, etc. in the way)? That’s what I would recommend looking at.
That line tells me that Google is more likely to crawl pages / web sites with a higher pagerank.
You will also notice he says we typically prefer urls with 1-2 parameters. Just to point out, that suggests that mod_rewrite of vBulletin is unecessary.
One question and answer caught my attention...
Q: “My sitemap has about 1350 urls in it. . . . . its been around for 2+ years, but I cannot seem to get all the pages indexed. Am I missing something here?”
A: One of the classic crawling strategies that Google has used is the amount of PageRank on your pages. So just because your site has been around for a couple years (or that you submit a sitemap), that doesn’t mean that we’ll automatically crawl every page on your site. In general, getting good quality links would probably help us know to crawl your site more deeply. You might also want to look at the remaining unindexed urls; do they have a ton of parameters (we typically prefer urls with 1-2 parameters)? Is there a robots.txt? Is it possible to reach the unindexed urls easily by following static text links (no Flash, JavaScript, AJAX, cookies, frames, etc. in the way)? That’s what I would recommend looking at.
That line tells me that Google is more likely to crawl pages / web sites with a higher pagerank.
You will also notice he says we typically prefer urls with 1-2 parameters. Just to point out, that suggests that mod_rewrite of vBulletin is unecessary.