Should we use Google's crawl delay setting?

[lzhao]

We’ve been noticing a huge uptick in Google’s spidering lately, and along with it a notable worsening of render times.

Yesterday, for example, Google spidered our site at a rate of 30:1 (google spider vs. organic traffic.)   So in other words, for every organic page request, Google hits the site 30 times.

Our render times have lengthened to an avg. of 2 seconds (and up to 2.5 seconds). Before this renewed interest Google has taken in us we were seeing closer to one second average render times, and often half of that.

A year ago, the ratio of Spider to Organic was between 6:1 and 10:1.

Is requesting a crawl-delay from Googlebot a viable option?

Our goal would be only to reduce Googlebot traffic, and hopefully improve render times and organic traffic.

Thanks,

Trisha

[RichardVaughan]

Unfortunately you can't change crawl settings for Google in a robots.txt file, they just ignore it. The best way to rate limit them is using custom Crawl settings in Google Webmaster Tools. (look under Site configuration > Settings)

You also might want to consider using your loadbalancer to direct Google (and other search engines) to a "condomised" group of servers (app, db, cache, search) thereby ensuring your users arent inadvertantly hit by perfomance issues caused by over zealous bot crawling.

[lzhao]

We're a publisher, which means that as an industry our normal render times are always at the top of the chart.  Ads are notoriously slow to load, and that's how we earn our keep.  These results are bad, though, even for publishing.

We're serving millions of uniques a month, on a bank of dedicated servers hosted off site, load balanced, etc.