Block search bots on staging server
-
I want to block bots from all of our client sites on our staging server. Since robots.txt files can easily be copied over when moving a site to production, how can i block bots/crawlers from our staging server (at the server level), but still allow our clients to see/preview their site before launch?
-
Robots.txt should be sufficient, but if you want to completely block them, here is a useful resource:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How old is 404 data from Google Search Console?
I was wondering how old the 404 data from Google Search Console actually is? Does anyone know over what kind of timespan their site 404s data is compiled over? How long do the 404s tend to take to disappear from the Google Search Console, once they are fixed?
Intermediate & Advanced SEO | | McTaggart0 -
One domain - Multiple servers
Can I have the root domain pointing to one server and other URLs on the domain pointing to another server without redirecting, domain masking or HTML masking? Dealing with an old site that is a mess. I want to avoid migrating the old website to the new environment. I want to work on a page by page and section by section basis, and whatever gets ready to go live I will release on the new server while keeping all other pages untouched and live on the old server. What are your recommendations?
Intermediate & Advanced SEO | | Joseph-Green-SEO0 -
Search queries results Wrong
Hey there, My website shows Wrong Search Queries in Google Search Console, Also Shows URLS which are not there in my website, Which Shows in crawl Errors. here i have attached Screenshot . http://prntscr.com/egzl88 Please Help me out how i can Deindex This type of URLs From Google index, & make My main pages crawl First in Google Search. because of this my website Ranking Also lost, Please any Expert can help out.. Thanx in advance.
Intermediate & Advanced SEO | | pooja.verify030 -
URL structure with broad search phrase but specific intent
My question is regarding some difficult URL structure questions in an online real estate marketplace. Our problem is that our customers search behavior is very broad, but their intent very narrow. For IRL examples go to objektia (dot) se. Example: Lease commercial space Stockholm Is a usual search query, wherein the user searches for the **broad category **commercial space, in the geography of Stockholm. The problem is that their intent is actually much more specific, since: Commercial space === [Office, Retail, Industrial, Storage, Properties] I have previously asked the forum for help regarding the placement of products in our URL-hierarchy, in which I got some good answers. We chose to go the route of alternative #3, ie placing our products (real estate listings), directly beneath their respective category (neighborhoods). https://moz.com/community/q/placement-of-products-in-url-structure-for-best-category-page-rankings Basically we chose to have the following URL structure: Structure: domain.se/category/subcategory/product Example: domain.se/Stockholm/suburb-of-stockholm/specific-listing-12 Now the question is, how do we deal with the **space type **modifier in our URL structure. Nobody wants to see retail space when they are after office space, so our current search page solution (category page) is the following: Structure: domain.se/space-type/neighborhood/sub-neighborhood All space types: domain.se/commercial-space/neighborhood/sub-neighborhood Specific space type: domain.se/office-space/neighborhood/sub-neighborhood Now, the problem with our current solution in combination with our intent to move our product pages into this hierarchy, is that every product page will be (and is today) linking towards the specific type category. Our internal link network would be built around type categories that are extremely relevant from a UX standpoint, but almost worthless (surprisingly) from an organic traffic standpoint. Also, every search page (category page) for each space type would be competing for the same search broad search phrase. The alternative is to place the type modifier at the end of the URL: Category page type at the end: domain.se/neighborhood/sub-neighborhood/type Listing page (product page), type at the end: domain.se/neighborhood/sub-neighborhood/street-address/type/listing-12
Intermediate & Advanced SEO | | Viktorsodd0 -
Robots.txt - Do I block Bots from crawling the non-www version if I use www.site.com ?
my site uses is set up at http://www.site.com I have my site redirected from non- www to the www in htacess file. My question is... what should my robots.txt file look like for the non-www site? Do you block robots from crawling the site like this? Or do you leave it blank? User-agent: * Disallow: / Sitemap: http://www.morganlindsayphotography.com/sitemap.xml Sitemap: http://www.morganlindsayphotography.com/video-sitemap.xml
Intermediate & Advanced SEO | | morg454540 -
Local Search For Multiple Locations With One Physical Address
I have a company that works in multiple locations but only has a physical address in one location. Is it possible to get this type of business listed in multiple locations? What is the most ethical way of doing this? Any help would be greatly appreciated. Christina
Intermediate & Advanced SEO | | ChristinaRadisic0 -
Changing Server IP Addresses. Should I be concerned?
Hello Mozers Our site has been on a dedicated server for about four years now. (no other sites, just ours on the server) I have made the decision to move it to a much better and faster server than the current server we are on for more than one reason. My big fear is Google will lose trust for my site because of the IP change. Ip's stay with the server at 1and1 they do not follow the website. So, I have done my due diligence and copied over all code and databases and have tested it completely to insure there are no issues when I change the DNS to point to the new server. Made sure 1and1 is giving me an IP that has never been used, I am Keeping the old server on until cached DNS records expire for it. Is there anything else I need to do to make sure I do not lose current rankings in Google? I have heard nightmare stories about making these kinds of changes but at this point for our site there is no turning back this is a change that must take place. Any pointers and advice would be much appreciated! Thanks!
Intermediate & Advanced SEO | | Robbie82991 -
URL blocked
Hi there, I have recently noticed that we have a link from an authoritative website, however when I looked at the code, it looked like this: <a <span="">href</a><a <span="">="http://www.mydomain.com/" title="blocked::http://www.mydomain.com/">keyword</a> You will notice that in the code there is 'blocked::' What is this? has it the same effect as a nofollow tag? Thanks for any help
Intermediate & Advanced SEO | | Paul780