Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
-
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare).
We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain.
If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains?
Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error!
Cheers,
Dave.
-
Hi Dave. A workflow checklist should really help with this as well. There are probably a few other items you'll catch by meeting with the others involved and getting everyone on the same page. Cheers!
-
Dave, I had exactly the same issue a month ago with being indexed on subdomains but was able to modify robots.txt in root domain swiftly enough to avoid real damage. Your main root robots.txt can override the subdomains. Simply disallow the subdomain in your robots file disallow: /subdomain url/robots.txt OR, if im not mistaken simply remove it altogether from the subdomain
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Naked domain redirection info
Hi Guys Been reading one or two posts about 'naked domains' v the 'www' derivative and was wondering... What is your opinion on this, is there a definitive benefit to your business in making the switch in terms of ranking? Apart from the Google released info, do you have any further recommended reading on this subject matter? Thanks in advance Daren
On-Page Optimization | | ITsoldSEO0 -
Ranking over someone else who has your branded domain name
Hello! I have a client who has been in business for a long time, but was very late to the game online. As a result, the branded domain for his business (for explanation purposes I'll call it "Houston Tan"*) was already taken when he decided he needed a website, however it was not being used. He approached the company that owned "houstontan.com" and they refused to sell it to him. Not only that, they turned around and opened a company and developed the website using his trademarked company name as one word instead of two, "HoustonTan." It was brought to court and the judge determined that since they created a new name by combining the two words, there was nothing that he could do. Still having to create a website for his company, he chose the domain "HoustonSunTan.com." Not sure who was advising him on that one. So now he has a domain name with only a partial match to his company name. As you would imagine, when you search Houston Tan, HoustonTan.com is number 1, 2 & 3, and HoustonSunTan.com is #4. My question is, do you think it is even possible for his company to overtake the top spot of Google? Or have you ever come across a situation like this and if so what worked for you? FYI: Even though it says Houston, the company is a global company in over 500 cities (with one 800 number unfortunately), so local SEO strategies wouldn't necessarily apply. *Names are made up to protect both parties 🙂
On-Page Optimization | | American.Made0 -
Can you reference http and https in a sitemap.xml?
I have a site where some pages (in Spanish) are https. The english pages are http. Can you tell me if it's okay to have both in the sitemap.xml?
On-Page Optimization | | RoxBrock0 -
Why do I hear that it can be a bad thing to have too many content pages?
My site has lots of long tail opportunity and so intention is to produce as much content as possible to target these keywords effectively. However, over the last 6 months I have come across numerous sources suggesting that Google can/does look down on a site having too many content pages. Is there any truth in this? Thanks.
On-Page Optimization | | Clicksjim0 -
Blocked By Meta Robots
Hi I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots. I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages. Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed? Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it? I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked. My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE. There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it. Blake
On-Page Optimization | | wayne10 -
Wordpress categories tags and robots.txt
I am relatively new at this and see a variety of people that seem to disagree on if you should block google from indexing category and tag pages through robot.txt or no-follow because of google viewing it as duplicate content. I tryst this communities answers over the web at large obviosly, so what do you all think? Thanks, Steven
On-Page Optimization | | sfmatthews0 -
Why is our site dropping in Domain Authority
Hi all, I have a question. I have been working SEO on our site www.betxpert.com for the last 4-6 weeks and I have been looking forward to seeing the new DA and PA numbers. However today, our DA is now 30...where it was 31 or 33 a couple of days ago. How come the decrease? I have only improved the site i should say... Hope someone can shed a little light on the issue... -Rasmus
On-Page Optimization | | rasmusbang0 -
Competitive Domain Analysis
In the competitve domian analysis my domain root (gpscity.com) is showing up as a 404 not found. My competition is showing up as 200. My site is live, index and viewable, but I'm wondering if this has an effect on my rankings? What can I do to test that it's not returning a 404 when the crawlers view my site?
On-Page Optimization | | tinyfactory0