How to block google robots from a subdomain
-
I have a subdomain that lets me preview the changes I put on my site.
The live site URL is www.site.com, working preview version is www.site.edit.com
The contents on both are almost identical
I want to block the preview version (www.site.edit.com) from Google Robots, so that they don't penalize me for duplicated content.
Is it the right way to do it:
User-Agent: *
Disallow: .edit.com/*
-
Thanks o much for your help!
-
Hi,
Probably without the www. so: site.edit.com/robots.txt because otherwise you would have a subdomain of a subdomain ;-). But the rest is perfect!
-
Thanks a lot for your answer, Martijn!
So just to make sure I got it correctly - this robots file URL should be:
?
Thanks a lot for your answer
-
Hi,
The Google Robots will look for the robots.txt in each individual root. So you need the robots.txt in the root of the subdomain not just the domain root. That's why its also possible to include a complete disallow in there and not just: .edit.com/* .
Example:
User-agent: *
Disallow: /Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Penalty and Adwords
Hi guys, I am wondering if the google manual penalty or penalty in general (because of bad backlink profile) also means that your website is blocked for Adwords? Thanks
Technical SEO | | barobijav0 -
What are the negative implications of listing URLs in a sitemap that are then blocked in the robots.txt?
In running a crawl of a client's site I can see several URLs listed in the sitemap that are then blocked in the robots.txt file. Other than perhaps using up crawl budget, are there any other negative implications?
Technical SEO | | richdan0 -
Block Domain in robots.txt
Hi. We had some URLs that were indexed in Google from a www1-subdomain. We have now disabled the URLs (returning a 404 - for other reasons we cannot do a redirect from www1 to www) and blocked via robots.txt. But the amount of indexed pages keeps increasing (for 2 weeks now). Unfortunately, I cannot install Webmaster Tools for this subdomain to tell Google to back off... Any ideas why this could be and whether it's normal? I can send you more domain infos by personal message if you want to have a look at it.
Technical SEO | | zeepartner0 -
How to Find all the Pages Index by Google?
I'm planning on moving my online store, http://www.filtrationmontreal.com/ to a new platform, http://www.corecommerce.com/ To reduce the SEO impact, I want to redirect 301 all the pages index by Google to the new page I will create in the new platform. I will keep the same domaine name, but all the URL will be customize on the new platform for better SEO. Also, is there a way or tool to create CSV file from those page index. Can Webmaster tool help? You can read my question about this subject here, http://www.seomoz.org/q/impacts-on-moving-online-store-to-new-platform Thank you, BigBlaze
Technical SEO | | BigBlaze2050 -
Remove Site from Google
How can I get my website out of google? I want all pages completely gone. Thanks!
Technical SEO | | tylerfraser0 -
Bandcamp subdomain
I have a website - www.weddingmusicproject.com that is doing quite well. However, we have all of the actual music listed on weddingmusicproject.bandcamp.com and have a number of very powerful pages there (bandcamp is a great product by the way). Is there a strong benefit to our domain's authority if we move our "bandcamp" site onto our own subdomain using their custom domain option - http://bandcamp.com/faq_custom_domains - Something like music.weddingmusicproject.com. I seem to think that it would increase our overall domain authority, but it wouldn't increase the number of inbound links or anything. If anything it would decrease the number of linking domains and bandcamp is quite a powerful site so those links would just turn into internal links. Thoughts? I know this is probably a basic concept, but I've thought it over a number of times and can't come to a conclusion.
Technical SEO | | deyobr0 -
Google Penalty?
Hi, I have recently been asked to help www.mycanvas.ie I have a feeling they have a google penalty. All their Google Keywords have literally dropped out of the Google SERP but they are still shown on Yahoo SERP. I recently did a site:www.mycanvas.ie and the pages are still in google index. The only thing that comes to mind is that the site owner submitted to 380 web directories over a period of 2 months with http://www.directorymaximizer.com/ do you think this could be causing the problem with google? Advise and suggestions are welcomed, thank you.
Technical SEO | | Socialdude0 -
Is this against google rules
Hi i am wanting to know if this is against google rules. I am building a website which will have lots of different sections and i wanted to know if you were allowed to have a new domain name pointing to a section of the site. so for example if i had a site with a domain name of manchester and then i wanted a section of the site to be called www.manchester.com/complimentary health I want to know if to help with traffic to the site and to have a better domain name, if it was allowed to have a new domain name pointing to that section of the site which could be called www.complimentaryhealth.com and have that pointing to the section. would love to hear your thoughts on this
Technical SEO | | ClaireH-1848860