Sitemap?

I have done quite a bit of reading here and elsewhere about the pros/cons of using a sitemap. I know that some have had good luck with them and some say that they have actually seen in decline in traffic (spiders). SO, my question to you is:

1. How do you build your sitemap?
a) Hardcode it
b) Plug-in
c) Software (generator)
d) vBulletin Archive for your site
e) none of the above

2. Do you submit to Google an xml or htm file or both?

Me personally, I use a software called Sitemap Generator and then clean up files so that it matchs my look and feel of my website.

Also, for those of you that have both a main site with a forum, do you bother creating a sitemap for both or do you use individual ones?

Thought I would try and get a discussion going for a dull Friday. :)

I was using Google Sitemaps for a while earlier this year. I never saw any benefit at all from them (although that was before the switch to vBulletin).

I do use a regular HTML sitemap on my main site - I think that's essential.

Howdy,

Can you give me/us some instruction on just how to create a HTML site map?

Have you tried vBSEO's auto generator...

http://www.vbulletin.org/forum/showthread.php?t=100435

I'm having a heck of a time trying to load that thing.

Thanks!

Lyte

I stole the basic design for mine from Apple. :o

See http://www.psychlinks.ca/pages/sitemap.htm.

Kinda picked a thread on Sitemaps. There were a few options :rolleyes:

I've been working hard on sitemaps ever since my Google problem with my homepage issue (http://www.vbulletin-faq.com/forum/showthread.php?t=5723&highlight=https)(still not resolved, so if anyone has any more suggestions...).

But I thought it might be an idea to really explore the sitemap issue. In no particular order, some of my thoughts and findings:

I still haven't worked out if vBSEO sitemaps has been ported for 3.6.x and available at vb.org - if anyone knows the answer to that?

I think once you've loaded a sitmap there might seem to be less bot activity, but that's because they aren't searching blindly anymore. And they do still seem to crawl stuff I didn't put in the sitemap such as the /tags/ stuff generated by the Zoints Tags hack, judging by the indexed pages for my site.

I have used the XML-Sitemap Generator (http://www.xml-sitemaps.com/) and once you've got the settings right, it works really well. For $10 I think it's worth it. I'll post my settings if anyone wants to go that route.

Realising that my homepage wasn't being crawled, I did a pretty intensive project to work out what Google had indexed. I was horrified at how many gaps there were, as well as how many really unimportant bits had been indexed. My goal in programming the XML sitemap generator was to get all custom pages, forum display pages, thread pages, and archive pages in the sitemap.

For this next section bear in mind I've got a bit under 500 public threads, the longest of which is 3 pages - and there are very few that are multi-page threads yet. It's kinda a nice sample size. Enough variety. Small enough to pick up patterns and not so large that you start to drown in info overload.

My first run I set no filters and crawled to 4 levels deep. This resulted in a sitemap with over 60 000 URLs. By the time I refined the filters to reach my goal, I was down to about 1100, and all the content I have as open was included at least once. Google's count on indexed pages has gone from under 500 (some of which was junk) to about 1070 last time I checked, in a timeframe of 2 weeks. And yes, some of the junk is still there :D and not all the pages on the sitemap appear as indexed - yet - particularly the older ones.

Most impressive is that when I add a custom content page, it shows up on Google within 24 hours of generating in the sitemap. New forum threads also seem to perform well. I've de-activated the daily/weekly/monthly option to see what happens. But Google seems to pick up on changes (new posts in a thread) pretty well by itself.

Somewhere here I read a minstrel post from Google talking about multiple URLs that produce the same page. The comment was that Google would pick its own URL to use if you didn't show a preference. I think Google does that anyway - going with the first in alphabetical order. This I get from behaviour on a 1 post thread. I find it is indexed as p= despite being t= in my sitemap. But as soon as the second post is added, it shows the t= and the p= disappears. Just an observation on the side.

I've still got to check the impact on Yahoo and MSN - I got indexed page counts from each before I started the sitemap project (and it wasn't a lot) but I didn't look too closely at just what was indexed at the time - other than noting that they both have my correct homepage indexed :cool:

From an SEO point of view, I was thinking it might be an idea to list each post individually. Any ideas on whether this would be a good or bad thing is appreciated.

In fact, any comments, suggestions, questions would be appreciated. It would be cool to get this sitemap issue put to bed, or at least thoroughly explored.