Well, this month has been interesting as far as traffic goes. MSNBOT has apparently gone apeshit on my site. For the first 26 full days of this month, they have hit my site over 56,000 times. That means they are averaging over 2,000 hits a day, with some truly exceptional days, like today.

Today, MSNBOT hit my site 6,261 times. On August 18th, they hit me 10,309 times. Most of their hits are coming at intervals of 1-2 seconds apart. I know when I was looking into writing a simple search spider for a former employer some time ago, everything I read about writing “polite” spiders indicated that you should only fetch one page from each site every minute at the most.

It seems like, as usual, greedy Microsoft wants to cheat a bit and get their search going ASAP, so of course they need to index faster, to everyone’s chagrin. Jeremy Zawodny mentioned MSNBOT on June 17th, and some of the comments there are very interesting.