Correct way to block search bots momentarily... HTTP 503?
-
Hi,
What is the best way to block googlebot etc momentarily? For example, if I am implementing a programming update to our magento ecommerce platform and am unsure of the results and potential layout/ file changes that may impact SEO (Googlebot continuously spiders our site)
How can you block the bots for like 30 mins or so?
Thanks
-
You can do that, but it is less specific on what you are actually doing with your server. The 503 and retry after lets the spiders know exactly what you are doing (no confusion). Thank you for the clever remark below.
-
Disregard mine, Clever was more... clever.. and beat me to it as well.
-
just disallow the root domain in your robots.txt file and when you're ready to let them back in edit your text file back to normal.
-
See the response here
http://moz.com/community/q/temporarily-shut-down-a-site
In short, the 503 is correct, you want to include a http header with a retry-after so it knows when to come back. Also, key to set this up on your robots.txt file as Google will key off of the status of this file. Once it sees that the robots.txt has a 503 it will wait until robots.txt shows a 200 again to then start crawling the entire site. Note that you still need to show the 503 on all pages, regardless.
Another option (that we use a lot on our larger sites) is that we have mirrored sites behind a load balancer. We will tell the load balancer to send traffic to www1,2 while we work on www3,4. When we have updated www3,4 we switch the load balancer to www3,4 and work on www1,2 and then when www1,2 are done we put them back into the mix on the load balancer. Makes it seamless for the users and for Google.
Cheers
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Significant "Average Position" dips in Search Console each time I post on Google My Business
Hi everyone, Several weeks ago I noticed that each Wednesday my site's Average Position in Search Console dipped significantly. Immediately I identified that this was the day my colleague published and back-linked a blog post, and so we spent the next few weeks testing and monitoring everything we did. We discovered that it was ONLY when we created a Google My Business post that the Average Position dipped, and on the 1st July we tested it one more time. The results were the same (please see attached image). I am 100% confident that Google My Business is the cause of the issue, but can't identify why. The image I upload belongs to me, the text isn't spammy or stuffed with keywords, the Learn More links to my own website, and I never receive any warnings from Google about the content. I would love to hear the community's thoughts on this and how I can stop the issue from continuing. I should note, that my Google My Business insights are generally positive i.e. no dips in search results etc. My URL is https://www.photographybymatthewjames.com/ Thanks in advance Matthew C0000OTrpfmNWx8g
White Hat / Black Hat SEO | | PhotoMattJames0 -
Scraped site, hijacked searches for business name.
Hello, I have a site that was scraped (possibly by a competitor's seo company), who then built links to the duplicate site. When people do a search for the name of the business the scraped site is all that comes up along with the usual third-party sites. They seem to take the site down and put it back up every couple of weeks to maintain the rankings in Google. Has anyone ever dealt with something like this? Any advice or recommendations would be appreciated. Search: LIC Dental Associates Scraped site: old-farmshow.net Legit site: licdentalassociates.com Thanks, Emery
White Hat / Black Hat SEO | | tntdental1 -
Homepage not ranking for branded searches after Google penalty removal
Hi all, A site I work on was hit with a manual action penalty some time ago for spammy links built by a former SEO agency. It was a partial match penalty so only affected some pages - most likely the homepage. We carried out a lot of work cleaning up links and disavowed suspicious links which we couldn't get removed. Again, most of these were to the homepage. The disavow file was uploaded to Google last Friday and our penalty was lifted this Tuesday. Since uploading the disavow file, our homepage does not show up at all for branded searches. I've carried out the obvious checks - robots.txt, making sure we're not accidentally noindexing the page or doing anything funky with canonicals etc and it's all good. Have any of you guys had a similar experience? I'm thinking Google simply needs time to catch up due to all the links we've disavowed and sitting tight is the best option but could do with some reassurance! Any past experiences or advice on what I might be missing would be great. Thanks in advance, Brendan.
White Hat / Black Hat SEO | | Brendan-Jackson1 -
URL disappeared from the search results
Hey folks, A URL on my webpage that has been climbing in search results ever since has suddenly completely disapeared from the search results and i'm absolutely stuck - no idea what the reason might be. It was ranked #11 for the targeted keyword, than it slightly started dropping down to #14 and #17 after which it completely disappeared, not only for specific targeted keyword, but also for exact name of the product. The URL has vanished from search results. I looked in search console, no particular errors or messages from Google. The only case I might come with is that many URLs are cannonicaly linked to the URL in matter, but i don't assume this might be the case. Does anyone have a suggestion what might the reason why the URL has completely vanished from the search results? Thank you a lot. The URL: http://chemometec.com/cell-counters/cell-counter-nc-200-nucleocounter/ Targeted keyword: 'cell counter'
White Hat / Black Hat SEO | | Chemometec0 -
Malicious bots
I was looking at some recommended keywords and felt sick to my stomach when I saw ilovevitaly.com search shell, resellerclub scam and a few more. | 2. | | 28(2.29%)ilovevitaly.com search shell | 0.00% | 0(0.00%) | 42.86% | 1.75 | 00:10:13 | 0.00% | 0(0.00%) | $0.00(0.00%) |
White Hat / Black Hat SEO | | BlueprintMarketing
| | 3. | resellerclub scam | I believe I have found the multiple IP addresses in which they're coming from and when I say many I mean I found 200 or so. There from different C blocks so they're very difficult to block easily without blocking legitimate traffic. I'm using a couple of different web application firewalls with the ability to block it pretty much anything. Does anyone have any device on doing this in a manner that might be more efficient than what I'm doing.I definitely do not want Google to think this is something that I did and penalize somebody this would be horrible. The site is going through Sucuri.net to be cleaned of any possible infection right now I do not know how this happened but zero day attacks are unfortunately a very real reality and unfortunately it could've been 1 million things. Thanks a million guys. I appreciate your help,
Tom0 -
Sitelinks Search Box impact for SEO
I am wondering how the relatively new sitelinks search box impacts the SEO rankings for a specific site or keyword combination - do you guys have any experience or bechmarks on this? Obviously it should help on getting more real estate on the SERP page (due to adding the search box), but do you also get extra goodwill and improved SERP position from adding it? Also, is the impact different on different type of terms, let's say single brand or category term such as "Bestbuy" (or "coupon") or a combination term "Bestbuy Apple" (or "Dixons coupon")? Thanks in advance!
White Hat / Black Hat SEO | | tjr0 -
What's the right way to gain the benefits of an EMD but avoid cramming the title?
Hi Guys, Say I'm (completely hypothetically) building weddingvenuesnewyork.com and right now I'm organizing the tags for each page. What's the best layout so that I can optimize for "wedding venues new york" as much as possible without it becoming spammy. Right now I'm looking at something like "Wedding Venues New York: Wedding Receptions and Ceremony Venues" for the title.. To get other strong keywords in there too. Is there a better layout/structure?.. And is having the first words of the title on the homepage the same as the domain name going to strengthen the ranking for that term, or look spammy to Google and be a bad move? This is a new site being built
White Hat / Black Hat SEO | | xcyte0
