Correct way to block search bots momentarily... HTTP 503?

[bjs2010]

Hi,

What is the best way to block googlebot etc momentarily? For example, if I am implementing a programming update to our magento ecommerce platform and am unsure of the results and potential layout/ file changes that may impact SEO (Googlebot continuously spiders our site)

How can you block the bots for like 30 mins or so?

Thanks

[CleverPhD]

You can do that, but it is less specific on what you are actually doing with your server.  The 503 and retry after lets the spiders know exactly what you are doing (no confusion).  Thank you for the clever remark below.  

[jesse-landry]

Disregard mine, Clever was more... clever.. and beat me to it as well.

[jesse-landry]

just disallow the root domain in your robots.txt file and when you're ready to let them back in edit your text file back to normal.

[CleverPhD]

See the response here

http://moz.com/community/q/temporarily-shut-down-a-site

In short, the 503 is correct, you want to include a http header with a retry-after so it knows when to come back.  Also, key to set this up on your robots.txt file as Google will key off of the status of this file.  Once it sees that the robots.txt has a 503 it will wait until robots.txt shows a 200 again to then start crawling the entire site.  Note that you still need to show the 503 on all pages, regardless.

Another option (that we use a lot on our larger sites) is that we have mirrored sites behind a load balancer.  We will tell the load balancer to send traffic to www1,2 while we work on www3,4.  When we have updated www3,4 we switch the load balancer to www3,4 and work on www1,2 and then when www1,2 are done we put them back into the mix on the load balancer.  Makes it seamless for the users and for Google.

Cheers