spiders hitting forum...

[onehost]

Just installed the forum a few days ago...google spiders are already hitting, but not in places where they should be crawling, they keep hitting calender..first they view it, I have 3 spiders right now that are trying to create an event..what the heck...? I even have the robots.txt - what are they doing, and why are they even there?

[Dave A]

The calendar is just full of links and pages, so spiders can get quite tied up in there.

[Mike54]

Review your robots.txt file, to be sure you're blocking everything that you want to block. I've seen several instances of people copying another's robots.txt and accomplishing nothing with it, because the first user had his forums installed in a /forum directory and the second user had his in a /forums directory.

It can take a few days for spiders to start obeying your robots.txt file. If they're up to playing nicely at all, that is.