Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
-
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare).
We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain.
If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains?
Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error!
Cheers,
Dave.
-
Hi Dave. A workflow checklist should really help with this as well. There are probably a few other items you'll catch by meeting with the others involved and getting everyone on the same page. Cheers!
-
Dave, I had exactly the same issue a month ago with being indexed on subdomains but was able to modify robots.txt in root domain swiftly enough to avoid real damage. Your main root robots.txt can override the subdomains. Simply disallow the subdomain in your robots file disallow: /subdomain url/robots.txt OR, if im not mistaken simply remove it altogether from the subdomain
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can we use "top 10" in title?
Hi All, This is a query regarding title tag.I have a listing page with list of hotels in each city. To improve CTR, I used "Top 10 hotels in Boston" as title even though I am showing 25 hotels per page. The page takes about "hotels" but I didn't use "top 10" term anywhere in the page. Is this called keyword stuffing? Should I use "top 10" somewhere in the page as my title says so? Thanks for your help!
On-Page Optimization | | Avinash_12340 -
Can't see the woods for the trees
Ok, I'm going be really cheeky here, so apologies in advance! We have this site www.simplychaise.co.uk and its not ranking anywhere in the top 50 for the term 'Chaise Longue' Moz has picked up a few issue with duplicate titles and a couple of 404's which I will fix shortly, It dosen't have any backlinks but I'm stumped as to why it isn't ranking higher than what it is, I'm not expecting 1st position but outside of the top 50 is something else. The site looks ok to me, but I think it might be a case of I've been looking at it so long that I can no longer see the wood for the trees. Could any kind hearted Mozer point me in the right direction here please, what am i missing? I'd greatly appreciate it, many thanks in advance for being so cheeky and well, useless! Jon
On-Page Optimization | | Jon-S0 -
Why isn't our site being shown on the first page of Google for a query using the exact domain, when its pages are indeed indexed by Google
When I type our domain.com as a query into Google, I only see one of our pages on the homepage, and it's in 4th position. It seems though, that all pages of the site are indexed by google when I type in the query "site:domain.com". There was an issue at the site launch, where the robots.txt file was left active for around two weeks. Would this have been responsible for the fact that another domain ranks #1 when we type in our own domain? It has been around a couple of months now since the site was launched. Thanks in advance.
On-Page Optimization | | featherseo0 -
Can rel="canonical" refer to another website page?
I want to republish the post from another website with their permission and want to abide by Google guidelines. Google guidelines is clear when you are using the same content at different parts of the same site however not when using it on another site in a legitimate way. Is there some way to use rel="canonical" refer to another website page of you are reproducing the content from same page?
On-Page Optimization | | h1seo0 -
Duplicate biographies across several domains?
Hey Gang, We've built a niche specific, manually edited legal community that is full of unique content. (While it's ultimately a directory, and that is often viewed as a bad word in these times, ours is a curated source that doesn't allow anyone and everyone to join.) We feel comfortable that it passes the sniff test post-panda/penguin, etc., and its doing rather well to date. The question we have is do we really need to create unique biographies for each of our legal members? Some of our competitors simply use the same bio information that the lawyer has on their own website and copies that to their site. The competitors I'm talking about are LARGE, well-respected, extremely successful folks, like FindLaw. Here's an example: Lawyer's website bio FindLaw website bio Both rank, etc., and the FindLaw code doesn't place any restrictions on their content regarding the bio., and it's an obvious exact match. I get that duplicate content is primarily a concern among one's own URL and the pages across a specific domain, using rel-canonical, etc., but what about two different domains that need to supply factual information that can't be altered? Is it anything we should worry ourselves with? Any tags we should insert in our code with regard to the bios? Thanks!!! Wayne
On-Page Optimization | | Wayne760 -
Can Your Site Get Penalized For Keyword Stuffing On An 'Untarged' Keyword?
My site has dropped since the EMD/Panda 20 roll out and I am looking for reasons why. I am looking at Keyword Stuffing as one potential problem. My web site is on the topic of WordPress Security with that being the main keyword I want to target. Now I can limit the number of occurrences of 'wordpress security' to below the recommended 15, but it is impossible to do this for 'wordpress' without severely compromising the user experience. I've got other content on topics such as WordPress Backup and WordPress Security Plugins etc, so obviously the word 'wordpress' is bound to appear frequently. Is there a risk that Google will penalize me for Keyword Stuffing on 'wordpress' and thus pull down the site or page for other keywords? Or would it simply mean I won't be able to rank for 'wordpress' (which I am quite happy about)? Thanks!
On-Page Optimization | | andersvin0 -
Hyphenated keyword rich domains - is a hyphenated .co.uk better than a non-hyphenatedlesser TLD such as .org?
Hey there, Would anyone be kind enough to share their experience of using keyword rich hyphenated domains. Are they as effective at tanking as non hyphenated domains? i.e would it be better to get a lesser non-hyphenated TLD such as .org for example rather than a hyphenated co.uk?
On-Page Optimization | | Wallander0 -
Cross Domain Duplicate Content
Hi My client has a series of websies, one main website and several mini websites, articles are created and published daily and weekly, one will go on a the main website and the others on one, two, or three of the mini sites. To combat duplication, i only ever allow one article to be indexed (apply noindex to articles that i don't wanted indexed by google, so, if 3 sites have same article, 2 sites will have noindex tag added to head). I am not completely sure if this is ok, and whether there are any negative affects, apart from the articles tagged as noindex not being indexed. Are there any obvious issues? I am aware of the canonical link rel tag, and know that this can be used on the same domain, but can it be used cross domain, in place of the noindex tag? If so, is it exactly the same in structure as the 'same domain' canonical link rel tag? Thanks Matt
On-Page Optimization | | mattys0