Robots.txt disallow subdomain
-
Hi all,
I have a development subdomain, which gets copied to the live domain. Because I don't want this dev domain to get crawled, I'd like to implement a robots.txt for this domain only. The problem is that I don't want this robots.txt to disallow the live domain. Is there a way to create a robots.txt for this development subdomain only?
Thanks in advance!
-
I would suggest you talk to the developers as Theo suggests to exclude visitors from your test site.
-
The copying is a manual process and I don't want any risks for the live environment. A Httphandler for robots.txt could be a solution and I'm going to discuss this with one of our developers. Other suggestions are still welcome of course!
-
Do you ftp copy one domain to the other? If this is a manual process the excluding the robots.txt that is on the test domain would be as simple as excluding it.
If you automate the copy and want code to function based on base url address then you could create a Httphandler for robots.txt that delivered a different version based on the request url host in the http request header.
-
You could use enviromental variables (for example in your env.ini or config.ini file) that are set to DEVELOPMENT, STAGING, or LIVE based on the appropriate environments the code finds itself in.
With the exact same code, your website would either be limiting IP addresses (on the development environment) or allow all IP addresses (in the live environment). With this setup you can also set different variables per environment such as the level of detail that is shown in your error reporting, connect to a testing database rather than a live one, etc.
[this was supposed to be a reply, but I accidentely clicked the wrong button. Hitting 'Delete reply' results in an error.]
-
Thanks for your quick reply, Theo. Unfortunately, this htpasswd will also get copied to the live environment, so our websites will get password protected live. Could there be any other solution for this?
-
I'm sure there is, but I'm guessing you don't want any human visitors to go to your development subdomain and view what is being done there as well? I'd suggest you either limit the visitors that have access by IP address (thereby effectively blocking out Google in one move) and/or implement a .htpasswd solution where developers can log in with their credentials to your development area (which blocks out Google as well).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens to crawled URLs subsequently blocked by robots.txt?
We have a very large store with 278,146 individual product pages. Since these are all various sizes and packaging quantities of less than 200 product categories my feeling is that Google would be better off making sure our category pages are indexed. I would like to block all product pages via robots.txt until we are sure all category pages are indexed, then unblock them. Our product pages rarely change, no ratings or product reviews so there is little reason for a search engine to revisit a product page. The sales team is afraid blocking a previously indexed product page will result in in it being removed from the Google index and would prefer to submit the categories by hand, 10 per day via requested crawling. Which is the better practice?
Intermediate & Advanced SEO | | AspenFasteners1 -
Unrelated subdomain hurts domain rankins?
Hi All, One of our subdomains has lot of content created by different users and mostly they are outgoing links from landing pages. Moreover the top ranking content is about "cigarettes" which is nowhere related to our niche. Will this hurt our domain rankings?
Intermediate & Advanced SEO | | vtmoz0 -
Redirecting main www. subdomain to new domain. Can you then create a new subdomain on the old domain?
Hi there, The scenario is this: We have been working on a rebrand and have changed the company name So, we want to redirect www.old-name.com to www.new-name.com However, the parent company is retaining the old brand name for corporate purposes So, in an ideal world, we'd be able to keep www.old-name.com active - but clearly that would sacrifice all of the authority built up over the years, so we do have to redirect the main www. subdomain in it's entirity. However - one suggested solution is to redirect www.old-domain.com to www.new-domain.com... but then create a new corporate subdomain: for example, business.old-domain.com business.old-domain.com will not be competing with the new site on any service/product related terms; it will only need to appear in SERPs for the company name I'd appreciate some thoughts on this, as I've not done this before or found any examples of anyone that has. Is that a massive risk in terms of sending a confusing message to Google? Thanks for your help
Intermediate & Advanced SEO | | edlondon0 -
Which one is better, a brand new subdomain or a second-level directory with PR 4
Hey, all SEOers! May I ask you a question about subdomain and second-level directory? Our website is about software, so we write many posts about how to use this software solve problems, and then use these posts to get ranks (we don't use the page of software to get ranks). And all the posts we wrote are listed under the second-lever directory, just like: www.xxx.com/support/ . But at this moment our boss want to list all the posts to the subdomain like support.xxx.com. By the way, the second-level directory is a page with PR 4, and the subdomain is brand new, even it doesn't exist now. So here is my question: should we list all the posts to support.xxx.com? If we choose to do like this, this will effect the speed of Google index, and we will take more time to build links for XXX.com and support.XXX.com? Any answer will be appreciated and thank you advance! to get rank instead of ranking the page of product,
Intermediate & Advanced SEO | | Vicky28850 -
Subdomain or folder for a section not focused on my core business
Hello there, I'm installing your analytics tool and it seems really great. I'm gonna use it for sure but I've a question that is more strategic and it's something the tool can't help me with 😛 I've a website active from 2008 and really well known in my country as a service website... we're like your "advisor" for utilities and insurances. The reason why is "savings" but really focused on utilities (broadband, gas, electricity) and check accounts or insurances. I’ve always used folders in my URLs instead of subdomains (for example www.site.com/section1 or www.site.com/section2 ). In this period I’m planning to open a new website section related to saving but not really close with what we really do in the rest of the website. This section is about coupons, vouchers and little offers. The problem is that with that section I’m going to write really a lot (a lot) of content trying to gain a lot of external links. It’s obvious that I already have a lot of contents about my core business and I’m going to write contents for original categories too. This section is anyway secondary for my business and my worry is that Google can identify me in the future as a website mainly focused on this new product. I’m really well indexed so I don’t want this decision to have any effect on my original situation. Finally the question 😛 Is it better to maintain for this section the same website structure with folders or indentify it as a subdomain to remark that it’s going to be like a totally different site with his dedicated news and all the rest? That’s why I’m evaluating a subdomain but I’m not really convinced cause subdomains can be considered as a different approach compared to original structure and of course using folder can be useful to gain root’s site rank. On the other hand, what can Google think about my core business? Thanks a lot for your help
Intermediate & Advanced SEO | | Uby850 -
Subdomain, subfolder or separate domains?
Hi Mozzers, We're in the process of re-developing and redesigning several of our websites, and moving them all onto the same content management system. At the moment, although the websites are all under the same brand and roughly the same designs, because of various reasons they all either live on a separate domain to the main website, or are on a subdomain. Here's a list of what we have (and what we're consolidating): Main site - http://www.frenchentree.com/ Property database - http://france-property.frenchentree.com/ (subdomain) Forum - http://www.france-forum-frenchentree.com/ (separate domain) Classified ads - http://www.france-classified-ads-frenchentree.com/ (separate domain) My question to you lovely people is: should we take this opportunity through the redevelopment of the CMS to put everything into subfolders of the main domain? Keep things as they are? Put each section onto a subdomain? What's best from an SEO perspective? For information - the property database was put onto a subdomain as this is what we were advised to do by the developers of the system. We're starting to question this decision though, as we very rarely see subdomains appear in SERPs for any remotely competitive search terms. Our SEO for the property database is fairly non-existent, and only ever really appears in SERPs for brand related keywords. For further info - the forum and classifieds were under a separate brand name previously, so keeping them on separate domains felt correct at that time. However, with the redevelopment of our sites, it seems to make more sense to either put them on subdomains or subfolders of the main site. Our SEO for the forum is pretty strong, though has dwindled in the last year or so. Any help/advice would be very much appreciated. Thanks Matt
Intermediate & Advanced SEO | | Horizon0 -
Robots
I have just noticed this in my code name="robots" content="noindex"> And have noticed some of my keywords have dropped, could this be the reason?
Intermediate & Advanced SEO | | Paul780 -
Subdomains vs. Subfolders for unique categories & topics
Hello, We are in the process of redesigning and migrating 5 previously separate websites (all different niche topics, including dining, entertainment, retail, real estate, etc.) under one umbrella site for the property in which they exist. From the property homepage, you will now be able to access all of the individual category sites within. As each niche microsite will be focused on a different topic, I am wondering whether it is best for SEO that we use subdomains such as category.mainsite.com or subfolders mainsite.com/category. I have seen it done both ways on large corporate sites (ie: Ikea uses subdomains for different country sites, and Apple uses subfolders), so I am wondering what makes the most sense for this particular umbrella site. Any help is greatly appreciated. Thanks, Melissa
Intermediate & Advanced SEO | | grapevinemktg0