Correct way to block search bots momentarily... HTTP 503?
-
Hi,
What is the best way to block googlebot etc momentarily? For example, if I am implementing a programming update to our magento ecommerce platform and am unsure of the results and potential layout/ file changes that may impact SEO (Googlebot continuously spiders our site)
How can you block the bots for like 30 mins or so?
Thanks
-
You can do that, but it is less specific on what you are actually doing with your server. The 503 and retry after lets the spiders know exactly what you are doing (no confusion). Thank you for the clever remark below.
-
Disregard mine, Clever was more... clever.. and beat me to it as well.
-
just disallow the root domain in your robots.txt file and when you're ready to let them back in edit your text file back to normal.
-
See the response here
http://moz.com/community/q/temporarily-shut-down-a-site
In short, the 503 is correct, you want to include a http header with a retry-after so it knows when to come back. Also, key to set this up on your robots.txt file as Google will key off of the status of this file. Once it sees that the robots.txt has a 503 it will wait until robots.txt shows a 200 again to then start crawling the entire site. Note that you still need to show the 503 on all pages, regardless.
Another option (that we use a lot on our larger sites) is that we have mirrored sites behind a load balancer. We will tell the load balancer to send traffic to www1,2 while we work on www3,4. When we have updated www3,4 we switch the load balancer to www3,4 and work on www1,2 and then when www1,2 are done we put them back into the mix on the load balancer. Makes it seamless for the users and for Google.
Cheers
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On the use of Disavow tool / Have I done it correctly, or what's wrong with my perception?
On a site I used GSA search engine ranker. Now, I got good links out of it. But, also got 4900 links from one domain. And, I thought according to ahrefs. One link from the one domain is equal to 4900 links from one domain. So, I downloaded links those 4900 and added 4899 links to disavow tool. To disavow, to keep my site stable at rankings and safe from any future penalty. Is that a correct way to try disavow tool? The site rankings are as it is.
White Hat / Black Hat SEO | | AMTrends0 -
Site Search external hosted pages - Penguin
Hi All, On the site www.myworkwear.co.uk we have a an externally hosted site search that also creates separately hosted pages of popular searches which rank in Google and create traffic. An example of this is listed below: Google Search: blue work trousers (appears on front page of Google) Site Champion Page: http://workwear.myworkwear.co.uk/workwear/Navy%20Blue%20Work%20Trousers Nearest Category page: http://www.myworkwear.co.uk/category/Mens-Work-Trousers-936.htm Could this be a penalisation or duplication factor? Could these be interpreted as a dodgy link factor? Thanks in advance for your help. Kind Regards, Andy Southall
White Hat / Black Hat SEO | | MarzVentures0 -
Has google done well with these search results?
I am struggling to grasp the new logic behind google, my understanding was that they wanted to return more related searches so that the search matched the results giving people exactly what they are looking for from trusted suppliers. However I work in the vacation rental niche and I have found that the individual long tail searches have started to become less valuable as they are no longer giving the exact property. Here is a screenshot of the top 10 results for the key phrase "10 bedroom villas in quinta do lago" Position 1 & 2 are good results and would be expected however the next 7 positions are completely not related to the search, yes it is quinta do lago. But I am looking specifically for a 10 bedroom villa, none of these pages offer 10 bedroom villas. I actually found my listing outside the top 20 and mine is a 10 bedroom villa in quinta do lago. Does anyone have anything that can enlighten me on this? Thanks Andy 0bqdRJi
White Hat / Black Hat SEO | | iprosoftware0 -
How do I write tags on a youtube video for a local Google search?
I've been reading into tags, and I would like to know what the best ways to do them for a local search are. Right now I have a title that reads similar to, "Keyword1 and Keyword2 in City X" Would I make a corresponding tag that reads "Keyword 1 and Keyword 2 in City X,"? Or would I do "Keyword 1," "Keyword 2," and, "City X," as separate tags? Thanks!
White Hat / Black Hat SEO | | OOMDODigital0 -
I am experiencing referrer spam from http://r-e-f-e-r-e-r.com/ (don't click) - What should I do?
It amazes me that every day in search marketing is filled with something new that I don't know or never heard of. Most of you are probably familiar with referrer spam, but I hadn't ever heard of it before. I am currently experiencing referral spam on my personal blog. What's the best way to get rid of this pest? Shall I ignore them? Block them in my robots.txt file? Use Google's Disavow? or should I just plain holler "Curse you referral spam people!!!" ? Thanks all!
White Hat / Black Hat SEO | | danatanseo0 -
Penguin Maybe? Ranking low for main term: Trying to find cause and correct
Hello, For nlpca(dot)com one of our main keywords is the term "NLP" We are ranking 25th for that term.Possible causes: 1. keyword stuffing on home page, though we need to use the term over and over again to describe ourselves. Also, competitors like nlpco(dot)com and nlpu(dot)com also mention "NLP" a lot 2. Backlink profile: see this spreadsheet. We have a lot of sites from other countries and many sitewides but all natural and almost all branded. Ou company names are NLP Institute of California, NLP California, and NLP and Coaching Institute. 3. nlpcacoach(dot)org is a sitewide footer link. So is iepdoc.nl. We're going to ask the first site to take our link down. 4. No "What is NLP" article. I think that might help. 5. Most of our 60 articles are posted on other sites. We author about 30 of them. I'm working on authorship via rel="author" and rel="me" links. There's usually 2 authors 6. Most of the title tags used to be 4 keywords separated by pipes -"|" I changed them all after the updates took the keyword "NLP" down. That's about all I can think of. What do we do or clean up?
White Hat / Black Hat SEO | | BobGW0 -
Can anyone recommend a Google-friendly way of utilising a large number of individual yet similar domains related to one main site?
I have a client who has one main service website, on which they have local landing pages for some of the areas in which they operate. They have since purchased 20 or so domains (although in the process of acquiring more) for which the domain names are all localised versions of the service they offer. Rather than redirecting these to the main site, they wish to operate them all separately with the goal of ranking for the specific localised terms related to each of the domains. One option would be to create microsites (hosted on individual C class IPs etc) with unique, location specific content on each of the domains. Another suggestion would be to park the domains and have them pointing at the individual local landing pages on the main site, so the domains would just be a window through which to view the pages which have already been created. The client is aware of the recent EMD update which could affect the above. Of course, we would wish to go with the most Google-friendly option, so I was wondering if anyone could offer some advice about how would be best to handle this? Many thanks in advance!
White Hat / Black Hat SEO | | AndrewAkesson0 -
Why Google still display search result so bad?
When I search this keyword Backlink คือ by Google browser(Google.co.th) then I saw these Domain that is spam keyword and worse content (Spin content and can not understand what it said) อํานาจเจริญ.dmc.tv/?p=19
White Hat / Black Hat SEO | | taradmkt
ฉะเชิงเทรา.dmc.tv/?p=28 พังงา.dmc.tv/?tag=backlink หนองคาย.dmc.tv/?p=97 ขอนแก่น.dmc.tv/?tag=backlink ชัยนาท.dmc.tv/?p=70 ตราด.dmc.tv/?tag=backlink etc As you can see the search result**.** My question is 1. How to tell Google to check these network 2. Why these network display Top 10 for 3 weeks!!!!! and after that they rank drop. 3. Why Facebook page rank on Google in the search result Please make me clear.0