Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
-
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare).
We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain.
If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains?
Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error!
Cheers,
Dave.
-
Hi Dave. A workflow checklist should really help with this as well. There are probably a few other items you'll catch by meeting with the others involved and getting everyone on the same page. Cheers!
-
Dave, I had exactly the same issue a month ago with being indexed on subdomains but was able to modify robots.txt in root domain swiftly enough to avoid real damage. Your main root robots.txt can override the subdomains. Simply disallow the subdomain in your robots file disallow: /subdomain url/robots.txt OR, if im not mistaken simply remove it altogether from the subdomain
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Magento Canonical & Default Robots Settings
Hello! I'm working with Magento 1.9 for an eCommerce site with several hundred products. Currently I understand it is best practices to use the Canonical tag, however I also have my default robots set to "Index, Follow". Will this cause an issue having product pages set to index, follow but also having a canonical tag included? What are some best practices regarding Magento default robots & canonical tags? Any help is appreciated.
On-Page Optimization | | BretDarby0 -
Can you reference http and https in a sitemap.xml?
I have a site where some pages (in Spanish) are https. The english pages are http. Can you tell me if it's okay to have both in the sitemap.xml?
On-Page Optimization | | RoxBrock0 -
Domain authority dropped a lot in last month
I thought I'd asked this earlier but it seems to have disappeared. My client's domain authority has hovered at around 42-46 for the past 6 months. But in the past month it has dropped 6 points. Organic search traffic is up, no keyword rankings have fallen, external links are up slightly, but not massively. Competitors remain roughly the same DA although theirs are only around 14/16 anyway. What should I be looking for as to the cause?
On-Page Optimization | | mfrgolfgti0 -
Can rel="canonical" refer to another website page?
I want to republish the post from another website with their permission and want to abide by Google guidelines. Google guidelines is clear when you are using the same content at different parts of the same site however not when using it on another site in a legitimate way. Is there some way to use rel="canonical" refer to another website page of you are reproducing the content from same page?
On-Page Optimization | | h1seo0 -
What does this mean on first step up setting up a campaign? "Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here."
I am BRAND new to this, and setting up my first campaign. I choose subdomain, and entered www.pdsaz.com. This is the message I receive: We have detected that the domain www.pdsaz.com and the domain pdsaz.com both respond to web requests and do not redirect. Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here.
On-Page Optimization | | cschwartzel0 -
Would Changing the Titles of Root Categories Be Bad?
I have researched some more effective keywords to change my root category titles to. I am wondering if it would be a bad idea to change these titles considering all the things that could go wrong. From what i'm gathering there are a LOT of things that can go wrong but at the same time these things do need to be changed sometime! Is this a good or a bad idea & why? What could go wrong? Should I try changing the category titles one at a time instead of risking every one of my keywords / category titles not working out in the serps right away?
On-Page Optimization | | Mike.Bean0 -
Forcing keywords into domain structure
Hi there, Over the last few years, I've seen people structuring their site so that their main content is all housed in a folder named after the site's primary keywords. For example, if I had some content about home insurance, normally naming conventions state that I might put the content at a URL such as: www.mydomain.com/home-insurance However, some sites, may change this structure to include their main keyword again in the URL string: www.mydomain.com/insurance/home-insurance The folder 'insurance' would normally hold the site's Sitemap to increase internal linking strategy too. I'd be really interested to hear whether anyone has seen any serious benefits from re-structuring their site in this way? What are your thoughts on this? Thanks,
On-Page Optimization | | theshortstack0 -
Domain authority
Wondering why my domain authority has dropped after about three months of trying to implement the best practices suggested on this site. Can anyone help?
On-Page Optimization | | MauiCoffee0