MSNBOT Going Berserk

Well, this month has been interesting as far as traffic goes. MSNBOT has apparently gone apeshit on my site. For the first 26 full days of this month, they have hit my site over 56,000 times. That means they are averaging over 2,000 hits a day, with some truly exceptional days, like today.

Today, MSNBOT hit my site 6,261 times. On August 18th, they hit me 10,309 times. Most of their hits are coming at intervals of 1-2 seconds apart. I know when I was looking into writing a simple search spider for a former employer some time ago, everything I read about writing “polite” spiders indicated that you should only fetch one page from each site every minute at the most.

It seems like, as usual, greedy Microsoft wants to cheat a bit and get their search going ASAP, so of course they need to index faster, to everyone’s chagrin. Jeremy Zawodny mentioned MSNBOT on June 17th, and some of the comments there are very interesting.

One thought on “MSNBOT Going Berserk

What do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s