Block search bots on staging server
-
I want to block bots from all of our client sites on our staging server. Since robots.txt files can easily be copied over when moving a site to production, how can i block bots/crawlers from our staging server (at the server level), but still allow our clients to see/preview their site before launch?
-
Robots.txt should be sufficient, but if you want to completely block them, here is a useful resource:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Suddenly keywords Disappeared from Google Search Results
Hello Guys Please help me, suddenly all of my site's keywords are disappeared from google search result, most of keywords are no.1 on google but today after 6pm i see the traffic decreasing and when i search my keywords there is no any keywords in search result. Only homepage keyword is showing. Please Help what is Happening with me.
Intermediate & Advanced SEO | | mianazeem4180 -
How to Improve Search Ranking from 10 to Top 5
Hi, I was just optimizing a page and have improved it from Page 2 -> Page 1 ; Position 15->10 for now by basic onpage seo edits. I want to understand how I would take it to next level by getting it to Top 5 or Top 3 results. My keyword and page are as follows (all checked in Google India (.in) ) Page - https://nirogam.com/ayurvedic-treatment-home-remedies-chikungunya/
Intermediate & Advanced SEO | | pks333
Keyword - ayurvedic treatment for chikungunya What steps can I take to go from 10 - Top 3 or Top 5 results here? I was checking the Rank tracker & moz Grader for this - got all ticks except adding keyword in URL. Would this be recommended of changing the URL after it's ranking so well to just add this keyword in the same?0 -
Only The Google Plus URL Appearing In Organic Search (No Images or Info)
Just a quick one, when I search for my brand in Google I see my normal listing but the Google Plus URL is appearing on the right hand side as a link on it's own. I see that other brands have there company info pulled in here. Is there anything I can do to make it appear like other brands have it, (with the images, text, etc.)? Thanks,
Intermediate & Advanced SEO | | the-gate-films0 -
Sitelinks in non-brand based organic search results
Hi all, I have a question for everyone. Sitelinks have been around for a while now & I've always seen them when the search is for a brand's name. However, today, when looking at the rankings for one of the campaigns we manage, we noticed there were sitelinks in the number #1 & #2 positions in Google (Australia) for the search term "Dance Costumes". Whilst both the companies have Dance Costumes in their title, so do all the other results & so I don't see why it warrants the sites to be relevant via their brand name.
Intermediate & Advanced SEO | | KBB_Digital
Note: The results are organic results, not paid results (where you can add sitelinks). Firstly, has anyone seen this before (screenshot attached)?
And secondly, is there markup/schema that allows you to do this (none that I know of)? danceCostumes-sitelinks.png0 -
Different website is shown when searched for content
Hello, Have any body experienced this situation, When i take a content from http://www.creativethemes.net e.g. from home page and search it on google, search results shows FMEextensions with that content. FME had never ever used that data, it can not be found in sources files either, also this data was never written and placed on FME site but it is shown with FME How can this happen and what do you suggest to resolve it?
Intermediate & Advanced SEO | | MozAddict0 -
Search Refinement URLs
My site is using search refinement and I am concerned about the URL adding additional characters when it's refined. My current URL is: http://www.autopartscheaper.com/Air-Conditioning-Heater-Parts-s/10280.htm and when someone chooses their specific year, make, and model then it changes to: http://www.autopartscheaper.com/Air-Conditioning-Heater-Parts-s/10280.htm?searching=Y&Cat=10280&RefineBy_7371=7708. Will this negatively affect SEO for this URL? Will the URL be counted twice? Any help would be great!
Intermediate & Advanced SEO | | BrandLabs0 -
Google Re-Index or multiple 301 Redirects on the server?
Over a year ago we moved a site from Blogspot that was adding dates in the URL's (i.e.. blog/2012/08/10/) Additionally we've removed category folders (/category, /tag, etc). Overall if I add all these redirects (from the multiple date options, etc) I'm concerned it might be an overload on the server? After talking with the server team they had suggested using something like 'BWP Google Sitemaps' on our Wordpress site, which would allow Google some time to re-index our site. What do you suggest we do?
Intermediate & Advanced SEO | | seointern0
