I was browsing through Google’s Webmaster Tools, and after submitting and verifying TYPELiFE, I noticed when I expanded the “tools” tab on the left, there was a menu option that said “set crawl rate” … hmmm, I wonder what this does?

A quick skim over the tool page shows me:

“Google collects documents on the web using Googlebot, our web crawler. You should leave this control at the Normal setting unless you are having trouble with the speed at which Googlebot is crawling your server.”

“We’ve detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn’t use too much of your server’s resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below.”

It also lets you know the current rate Googlebot is indexing your website.

If you think you can handle it, choose a new crawl speed! I set mine to faster, now I’m just waiting to see the results!

This site is currently set at a Faster crawl rate. This rate will return to Normal on Mar 10, 2008.