Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Block search bots on staging server

Intermediate & Advanced SEO

597

BlueView1301 last edited by

I want to block bots from all of our client sites on our staging server. Since robots.txt files can easily be copied over when moving a site to production, how can i block bots/crawlers from our staging server (at the server level), but still allow our clients to see/preview their site before launch?
1 Reply Last reply
Reply Quote 0
TakeshiYoung last edited by

Robots.txt should be sufficient, but if you want to completely block them, here is a useful resource:

http://www.seomoz.org/ugc/blocking-bots-based-on-useragent
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Robots.txt blocked internal resources Wordpress

Hi all, We've recently migrated a Wordpress website from staging to live, but the robots.txt was deleted. I've created the following new one: User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Allow: /wp-admin/admin-ajax.php However, in the site audit on SemRush, I now get the mention that a lot of pages have issues with blocked internal resources in robots.txt file. These blocked internal resources are all cached and minified css elements: links, images and scripts. Does this mean that Google won't crawl some parts of these pages with blocked resources correctly and thus won't be able to follow these links and index the images? In other words, is this any cause for concern regarding SEO? Of course I can change the robots.txt again, but will urls like https://example.com/wp-content/cache/minify/df983.js end up in the index? Thanks for your thoughts!
Intermediate & Advanced SEO | | Mat_C

2
Block web archieve/way back machine

Hi i want to block web archive/wayback machine from indexing my site and creating a record of it on their database. Any ideas on how to do this? Cheers,
Superpak
Intermediate & Advanced SEO | | Mikey008

2
What URL parameter settings in GWT to choose for search results parameter?

Hello,we're about to disallow search results from crawling in robots.txt, but in GWT we have to specify URL parameters. URLs with 'search' parameter look like these: http://www.example.com/?search=keyword So in GWT we're setting the following parameter: search Question, what settings to set for it?
Intermediate & Advanced SEO | | poiseo

0
Should You Use 301 Redirects When Switching To A Secure SSL Server?

Hi, our client has switched from a non-secure server to a secure (SSL) server.. but the non secure pages still exist, i.e. http://www.stainlesshandrailsystems.co.uk/balustrade-systems.html (non-secure)
https://www.stainlesshandrailsystems.co.uk/balustrade-systems.html (secure) We assumed that we should 301 redirect the http pages to the new https pages using the following htaccess rule; RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.yoursite.com/$1 [R,L] HOWEVER! both of the above pages show the same Page Authority (PA) and Pagerank (PR).. does this mean that they are being seen as the same page, do we really need to employ 301 redirects? Many thanks in advance, much appreciated. 🙂 Lee
Intermediate & Advanced SEO | | Webpresence

1
Problem with description on Google search results.

A few months ago I changed the description of one of the pages on my site.
And I noticed that Google does not display the entire description of his search results. Description page is: "Get yourself a personalized name necklace, we offer a huge range of silver, gold and gold plated name necklaces." And Google only shows this line: "Get yourself a personalized name necklace, we offer a huge ... " Did someone have an idea why is that? 2EPSLGX.png
Intermediate & Advanced SEO | | Tiedemann_Anselm

0
Alternative links in the search results.

Hello, This is a short question Please look at this SERP screenshot: http://imgur.com/1EMen Who do they get the other links under their results. Cornel
Intermediate & Advanced SEO | | Cornel_Ilea

0
How to handle web server downtime?

We have a client who is taking their web server down Saturday morning from 1am - 7am for planned maintenance. Initially, we thought to have all requests return a 503 (service unavailable) response but the web server itself will be down so we are not able to have it return any response codes. Updating the DNS on the registrar will have too much lag time while it propogates out so we aren't sure exactly how to handle this. I had thought possibly of using a second DNS, or a service like DynDNS but that seems like a large amount of effort to set up just for some planned downtime. I have to imagine that Google understands planned website/server downtime every once in a great while. This client has pretty good rankings for some incredibly competitive terms so we want to do all that we can to make sure those rankings are preserved. What are some other potential solutions? We could totally just be overthinking this but we'd rather be safe than sorry... Thanks in advance!
Intermediate & Advanced SEO | | MichaelWeisbaum

0
Is User Agent Detection still a valid method for blocking certain URL parameters from the Search Engines?

I'm concerned with the cloaking issue. Has anyone successfully implemented user agent detection to provide the Search engines with "clean" URLs?
Intermediate & Advanced SEO | | MyaRiemer

0