Block search bots on staging server
-
I want to block bots from all of our client sites on our staging server. Since robots.txt files can easily be copied over when moving a site to production, how can i block bots/crawlers from our staging server (at the server level), but still allow our clients to see/preview their site before launch?
-
Robots.txt should be sufficient, but if you want to completely block them, here is a useful resource:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New blog on a separate server to the main website?
We have a potential client who operates a jobs board in a niche sector in the UK. They want to start a blog but don't want to set it up on the same server as the main jobs site. Discussion started around Wordpress, and their preference is for the WP.com hosted version in a directory or subdomain of the TLD. Our concerns are around the different locations of the two sites (impact of two diff server locations and IP addresses?) but also the limitation of WP.com to interlink the two sites enough that they provide a decent customer experience. Thoughts, musings, advice - all welcome! Tks
Intermediate & Advanced SEO | | AB-Marketing0 -
How to compete against search terms that use geo-modifiers?
I should start by saying we are new to SEO. We are introducing new “cycling tours” in new destinations and we are looking for a strategy to combat geo-modified keyword searches. When people search for “cycling tours” they will anchor their search with a geo-modifier such as “cycling tours France” or “cycling tours Italy”. Based in Australia we are keen to communicate to Australians searching for international cycling tours there are new Australian options that they may wish to consider. The geo-modifiers required to find our tours (“eyre peninsula” and “carnarvon gorge”) are currently not on the cycling communities radar. For example to find one of our new tours you need to use “cycling tours eyre peninsula” or “cycling tours carnarvon gorge”. Currently the only solution we have found to let people know about our new tours is by word of mouth. Is there an SEO solution?
Intermediate & Advanced SEO | | Chook10 -
Search engine blocked by robots-crawl error by moz & GWT
Hello Everyone,. For My Site I am Getting Error Code 605: Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag, Also google Webmaster Also not able to fetch my site, tajsigma.com is my site Any expert Can Help please, Thanx
Intermediate & Advanced SEO | | falguniinnovative0 -
How do we preserve images in google search after CMS migration?
Hi Folks we are about to migrate to a new CMS (bigcommerce/volusion type of thing) are are advised that we will preserve our google love for our old URLS with 301 re-directs. OK but what about images that show in search (we have a lot of our images show up high in relevant google image search) will this method work the same or should we do something else to keep the image benefits? many thanks Tom
Intermediate & Advanced SEO | | tomnivore0 -
Will multiple domains from the same company rank for the same keyword search?
I'm trying to convince people that we need good marketing reasons for starting multiple domains, as it will be more difficult to rank multiple sites. Does anyone know if Google actively discourages multiple domains from the same company appearing in the search results for the same keyword? We are creating a separate content website which is related to an existing company website. Would you agree that is best to have these sites on one domain with the content site on a sub-domain perhaps? I'm worried about duplication of effort and cross-keyword targeting in particular. These sites would not have duplicate content.
Intermediate & Advanced SEO | | RG_SEO0 -
Blog posts not showing in serps for exact match title search
hi- my first client ranks #1 for the exact phrase of each blog post title the 2nd client doesnt rank anywhere when i search for the exact post title 2nd client has robots.txt User-agent: *
Intermediate & Advanced SEO | | Ezpro9
Disallow: /wp-admin/
Disallow: /wp-includes/ so that shouldnt noindex any posts right? his site ranks for many kw's - but oddly none of his blog posts are anywhere to be found - i dont mean for a kw search - i mean for searching for the entire title he doesnt rank anywhere in first 5 pages for any of 6-7 posts i checked any idea what could cause this? thanks0 -
Blog not showing up when searching for exact post/META titles
I am working on a blog http://www.possessionista.com which is a very popular fashion blog. It is very well established with a 100% natural link profile and zero spammy stuff. The blog ranks #1 for random fashion terms like "kourtney kardashian cat eye sunglasses" and "emily maynard boots". The problem I am experiencing is that none of the actual titles of her posts or any of the content in the post results in her blog showing up if searched. EX: http://www.possessionista.com/2011/10/pippa-middletons-zip-jacket.html When you search "Pippa Middleton's Zip Jacket" on google her blog is nowhere to be found. Try searching allintitle:"Pippa Middleton's Zip Jacket" and she's nowhere to be found either. Even search "The other day, I met with my friend Kiran for our monthly mutual admiration society" on google and she's nowhere to be found even thoguh this is a unique snippet from her post. This post is already indexed and cached with the above mentioned details. i've also tested dozens of older posts as well. Same issue. You can actually do this to see a more clear picture: Do a google search for: allintitle:Bachelorette Fashion: Episode 2 - Ashley Hebert Brown site:possessionista.com That will bring up her blog which means google recognizes that the phrase is in her META title. Now do a google search for: allintitle:Bachelorette Fashion: Episode 2 - Ashley Hebert Brown without the site: included. She does not pop up but other people do. I did find that she had a duplicate title tag for a few weeks, but I've fixed that. Her posts used to pop up #1 when you search the title, but now obviously not. I am kind of at a loss and have tried a bunch of options with no success. Oh, one other thing is that some people do scrape her content, but only a few like maybe 10 and they've always been doing it even when she used to rank for her own post titles. Have you guys experienced this issue? Do you have any ideas of how to fix it?
Intermediate & Advanced SEO | | modparent0