How to block google robots from a subdomain
-
I have a subdomain that lets me preview the changes I put on my site.
The live site URL is www.site.com, working preview version is www.site.edit.com
The contents on both are almost identical
I want to block the preview version (www.site.edit.com) from Google Robots, so that they don't penalize me for duplicated content.
Is it the right way to do it:
User-Agent: *
Disallow: .edit.com/*
-
Thanks o much for your help!
-
Hi,
Probably without the www. so: site.edit.com/robots.txt because otherwise you would have a subdomain of a subdomain ;-). But the rest is perfect!
-
Thanks a lot for your answer, Martijn!
So just to make sure I got it correctly - this robots file URL should be:
?
Thanks a lot for your answer
-
Hi,
The Google Robots will look for the robots.txt in each individual root. So you need the robots.txt in the root of the subdomain not just the domain root. That's why its also possible to include a complete disallow in there and not just: .edit.com/* .
Example:
User-agent: *
Disallow: /Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What to do about this subdomain for SEO?
This is a bit of an unusual structure and I'm having difficulty explaining the question so pardon my being a 'noob', haha. The website I'm working on has some content under Forums that is hosted on another domain. The main website is https://yournorthside.org.au/ and if you select under the main Nav > Forums > Lived Experience it will take you to https://yournorthside.saneforums.org/t5/Lived-Experience-Forum/ct-p/lived-experience-forum. So it's as if it's a subdomain. (notice even the appearance of the main menu changes, weird) Apparently, saneforums.org has a requirement for that content to be on that subdomain. So therefore it's not part of my sitemap and now crawled or indexed. My question is is this structure okay? What are the implications for SEO? Should I be looking to implement some type of no follow link or something? Or is it actually beneficial in terms of all their content gives us 'link juice'? Can you link me to any resources / articles that give further insight?
Technical SEO | | kelseyc0 -
What's going on with google index - javascript and google bot
Hi all, Weird issue with one of my websites. The website URL: http://www.athletictrainers.myindustrytracker.com/ Let's take 2 diffrenet article pages from this website: 1st: http://www.athletictrainers.myindustrytracker.com/en/article/71232/ As you can see the page is indexed correctly on google: http://webcache.googleusercontent.com/search?q=cache:dfbzhHkl5K4J:www.athletictrainers.myindustrytracker.com/en/article/71232/10-minute-core-and-cardio&hl=en&strip=1 (that the "text only" version, indexed on May 19th) 2nd: http://www.athletictrainers.myindustrytracker.com/en/article/69811 As you can see the page isn't indexed correctly on google: http://webcache.googleusercontent.com/search?q=cache:KeU6-oViFkgJ:www.athletictrainers.myindustrytracker.com/en/article/69811&hl=en&strip=1 (that the "text only" version, indexed on May 21th) They both have the same code, and about the dates, there are pages that indexed before the 19th and they also problematic. Google can't read the content, he can read it when he wants to. Can you think what is the problem with that? I know that google can read JS and crawl our pages correctly, but it happens only with few pages and not all of them (as you can see above).
Technical SEO | | cobano0 -
Web Page Dropped Out of Google?
One of our web pages seems to have completely dropped out of Google after featuring on page 1 for a number of years. It can't be a site wide issue as all other web pages are performing as normal. The page is http://www.contractormoney.com/income-protection/ and the key phrase it was performing well for was 'contractor income protection'. Any ideas??
Technical SEO | | Pete40 -
Using Google Adwords is good?
I heard about that if you using adwords, google drops your ranking a little bit. Because of you already pay money for results. I think that is reasonable.
Technical SEO | | umutege0 -
Google (GWT) says my homepage and posts are blocked by Robots.txt
I guys.. I have a very annoying issue.. My Wordpress-blog over at www.Trovatten.com has some indexation-problems.. Google Webmaster Tools data:
Technical SEO | | FrederikTrovatten22
GWT says the following: "Sitemap contains urls which are blocked by robots.txt." and shows me my homepage and my blogposts.. This is my Robots.txt: http://www.trovatten.com/robots.txt
"User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/ Do you have any idea why it says that the URL's are being blocked by robots.txt when that looks how it should?
I've read a couple of places that it can be because of a Wordpress Plugin that is creating a virtuel robots.txt, but I can't validate it.. 1. I have set WP-Privacy to crawl my site
2. I have deactivated all WP-plugins and I still get same GWT-Warnings. Looking forward to hear if you have an idea that might work!0 -
Blog on a subdomain vs subfolder?
Hi, Does anyone have data to show that a subfolder is better than a subdomain for a blog? From what I've read, it sounds like both are a viable option but you choose subdomain if you want to build your blog as a distinct entity. Do you get ranked more quickly with a subfolder? Do you see X% more lift? Has anyone tested or seen tests around this subject? Any input is appreciated! Thanks in advance.
Technical SEO | | sportstvjobs0 -
Subdomains
Hi, I have recently started working in-house for a company and one site development was started and completed just as I joined. A new area of the site has been developed, but the developers have developed this new section in php, which cannot be hosted on the windows server the site is running on (they tell me, is this correct?) They want to add the new section as a subdomain - http://newarea.example.co.uk/ whereas I would have preferred the section added as a new subfolder. I plan to ensure that future developments to not have this problem, but is the best solution to work with the subdomain (in this instance it may not be too bad as it is a niche area of the site), or can I redirect the pages hosted on the sub-domain to a subfolder, and is this recommended? Thanks for your time.
Technical SEO | | LSLPS0