Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
-
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare).
We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain.
If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains?
Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error!
Cheers,
Dave.
-
Hi Dave. A workflow checklist should really help with this as well. There are probably a few other items you'll catch by meeting with the others involved and getting everyone on the same page. Cheers!
-
Dave, I had exactly the same issue a month ago with being indexed on subdomains but was able to modify robots.txt in root domain swiftly enough to avoid real damage. Your main root robots.txt can override the subdomains. Simply disallow the subdomain in your robots file disallow: /subdomain url/robots.txt OR, if im not mistaken simply remove it altogether from the subdomain
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can you transcribe your youtube video to just google?
YouTube videos have been very effective for us on our site. The only problem is they have taken part of a fair amount of our content so to google our pages may seem to have minimal content. Transcribing each video and including that on the page would take away from the look of the page and clutter it. Is there a way to let google know the content without having to include it on the page? You can find an example page at https://www.waikoloavacationrentals.com/halii-kai-rentals/
On-Page Optimization | | RobDalton0 -
Can anyone see what hurt me in my traffic
http://www.freescrabbledictionary.com/ I need some fresh eyes on this traffic of mine. I have compared the dates to when updates have been rolled out (panda, penguin, page layout, top heavy, etc...) and I cant match these dates with any of them. Can anyone maybe shed some light to my traffic drops and recovery then traffic drop again and see if there is a correlation with any major updates? Dropped Jan 22 2014 Dropped April 4 2014 Recovered August 25 2014 Dropped September 18 2014 google-analytics.jpg
On-Page Optimization | | cbielich0 -
Subdomain vs. Root Domain design
I know that there are penalties if you have a navigation item in your categories that links to another site (root domain) that yields the same design as the original link. Is there any negative SEO implications with having a subdomain with the same design as the root domain i.e. root.com vs. store.root.com And the navigation functionality within will have links to both store.root.com as well as root.com directories? Thanks in advance.
On-Page Optimization | | customerparadigm.com0 -
Hyphens in Domain Name
I have a client who is a business broker. I have just begun working with them, and they are trying to determine the best domain name to use - they have several. business-broker-alabama.com vs businessbrokeralabama.com Which one of these has more value? I know that search engines have somewhat devalued keywords. The first one probably has a little more SEO value, but is going to be a drag in terms of marketing -- saying "business hyphen broker hyphen alabama" is going to get old for them. Thanks!
On-Page Optimization | | csmithal0 -
Using meta robots 'noindex'
Alright, so I would consider myself a beginner at SEO. I've been doing merchandising and marketing for Ecommerce sites for about a year and a half now and am just now starting to attempt to apply some intermediate SEO techniques to the sites I work on so bear with me. We are currently redoing the homepage of our site and I am evaluating what links to have on it. I don't want to lose precious link juice to pages that don't need it, but there are certain pages that we need to have on the homepage that people just won't search for. My question is would it be a good move to add the meta robots 'noindex' tag to these pages? Is my understanding correct that if the only link on the page is back to the homepage it will pass back the linkjuice? Also, how many homepage links are too many? We have a fairly large ecommerce site with a lot of categories we'd like to feature, but don't want to overdo the homepage. I appreciate any help!
On-Page Optimization | | ClaytonKendall0 -
Robots.txt: excluding URL
Hi, spiders crawl some dynamic urls in my website (example: http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/ + http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/open=true) as different pages, resulting duplicate content of course. What is syntax for disallow these kind of urls in robots.txt? Thanks so much
On-Page Optimization | | anakyn0 -
What reasons exist to use noindex / robots.txt?
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
On-Page Optimization | | digitalstream0 -
301 redirect (www.domain.com/index to www.domain.com)
Hello, Please let me know what are the exact right steps in order to get rid of the duplicate content issues related with: www.domain.com/index.html same as www.domain.com without creating an infinite loop. Do you have a step by step guide posted within seomoz including 301 redirect for non www to www for all urls and index.whatever to main domain name without going into a infinite loop ? btw how to you spot the loop ? is it obvious like never ending refresh of the home page ? thanks a lot !
On-Page Optimization | | eyepaq2