Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
-
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare).
We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain.
If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains?
Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error!
Cheers,
Dave.
-
Hi Dave. A workflow checklist should really help with this as well. There are probably a few other items you'll catch by meeting with the others involved and getting everyone on the same page. Cheers!
-
Dave, I had exactly the same issue a month ago with being indexed on subdomains but was able to modify robots.txt in root domain swiftly enough to avoid real damage. Your main root robots.txt can override the subdomains. Simply disallow the subdomain in your robots file disallow: /subdomain url/robots.txt OR, if im not mistaken simply remove it altogether from the subdomain
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can a page rank for keywords that it does not have on it?
I have a client that is ranking in the top 10 for several keywords on their homepage. Their site has no purposeful SEO in it, there is barely any text on the homepage at all and none of the text are the keywords it is ranking for.
On-Page Optimization | | woodchuckarts2 -
Duplicate ecommerce domains and canonical
Hi everybody! I'd like to discuss the SEO strategy I've thought regarding a client of mine and ask for help because it's a serious case of duplicate content. There is a main website (the business model one) where he compares the cost of medicines in several pharmacies, to show the cheapest shopping cart to the customer. But the shopping has to been made in another domain, within the selected pharmacie, because my country's law in Europe says that is compulsory to sell the medicines only on the pharmacy website. So my client has started to create domains, one for each pharmacy, where the differences between them are only some products, the business information of the pharmacy and the template's colour. But all of them shares the same product data base. My aim is to rank the comparing website (it contains all the products), not each pharmacy, so I've started to create different content for this one. Should I place rel=canonical in the pharmacies domains t the original one? For instance: www.pharmacie1.com >> www.originaltorank.com www.pharmacie2.com >> www.originaltorank.com www.pharmacie1.com/product-10 >> www.originaltorank.com/product-10 I've already discuss the possibilities to focus all the content in only one website, but it's compulsory to have different domains in order to sell medicines By the way, I can't redirect 301 because I need these websites exist for the same reason (the law) He is creating 1-3 new domains every week so obviously he has had a drop in his SEO traffic that I have to solve this fast. Do you think the canonical will be the best solution? I dont want to noindex these domains beacuse we're creating Google Local pages for each one in order to be found in their villages. Please, I'll appreciate any piece of advice. Thanks!
On-Page Optimization | | Estherpuntu0 -
Can the Lightboxes on My Site be Crawled?
I'm trying to optimize my site, but I have lightboxes and I don't know if they are visible to the search engines. If they aren't, could you suggest something that I could do? THANK YOU so much!!!!! My site is lymphexpo.com
On-Page Optimization | | bosleypalmer0 -
Unique Geo-targeted domains v. simple landing pages
I have a client with multiple locations in a central area. Is it better to purchase unique geo-targeted domains i.g. DowntownTopekaPlumber.com and MidtownTopekaPlumber.com and create area specific splash pages with links back to the main site -OR - is it better to simply create landing pages on the main domain - www.GoodGuyPlumbing.com/DownttownTopekaPlumber?
On-Page Optimization | | SearchParty0 -
I noticed that my company owns a few exact match domains that are relevant to our business. Is it worth doing anything other than just redirecting them to our homepage?
It turns out we own a few exact match domains that are relevant to our business. Is it worth doing anything other than just redirecting? (We have owned these sites for a long time, over 15 years.) The keywords are moderate to low in traffic and moderate to high in competitiveness. (Right now they 301 to our main site.) I know exact match is not as valuable as it used to be and that there are mixed opinions about microsites...
On-Page Optimization | | Linda-Vassily0 -
Can I add a paragraph of blog content to a product page?
I have a great blog post about one of my products, which means there's some copy in the blog that I would like in the product page. It would constitute perhaps 5% of the copy on the product page if I copied it.
On-Page Optimization | | Brocberry0 -
Best practice for Meta-Robots tag in categories and author pages?
For some of our site we use Wordpress, which we really like working with. The question I have is for the categories and authors pages (and similiar pages), i.e. the one looking: http://www.domain.com/authors/. Should you or should you not use follow, noindex for meta-robots? We have a lot of categories/tags/authors which generates a lot of pages. I'm a bit worried that google won't like this and leaning towards adding the follow, noindex. But the more I read about it, the more I see people disagree. What does the community of Seomoz think?
On-Page Optimization | | Lobtec0 -
Purchased domain
We purchased the existing domain www.LancasterPA.com two years ago to promote local businesses. While individual pages within the site are ranking well, we can't get the home page to rank at all in Google. Would there be anything in the history of the domain that could be standing in the way? Or, what else could we check? Our other regional websites are ranking really well. Thanks.
On-Page Optimization | | GordyH0