I happened to be looking through my awstats output for this month when I noticed something I found a little odd, or at least worth mentioning. I’ve been examining these types of logs for years now and there’s so much data there that perhaps I’ve missed it before, but in the list of search engine bot statistics, I noticed an interesting relationship. So far for the month of March, Googlebot has hit my site 409 times for a total bandwidth usage of 3.10MB. Yahoo has hit my site 719 times for a total bandwidth of 6.39MB. Last, but certainly not least is the MSNBot coming in with only 590 hits, but a whopping bandwidth consumption of 11.89MB! Now, there are a number of others in my list, that each consume far less than these big three. Looking back over the past several months’ data, Google and Yahoo appear to be rather even in this measurement, but MSNBot is the clear leader with sometimes 15 times the bandwidth consumption! I don’t exactly know for certain if it’s fair to interpret these numbers this way, but it occurred to me that Microsoft is trying extra hard to beat Google, and perhaps this increased bot activity is evidence of it. Anyway, just makes you wonder how well it’s going for them. ;)
I’d be interested to know if anyone else has seen this activity or has a better explanation for it.