Robots.txt disallow subdomain
-
Hi all,
I have a development subdomain, which gets copied to the live domain. Because I don't want this dev domain to get crawled, I'd like to implement a robots.txt for this domain only. The problem is that I don't want this robots.txt to disallow the live domain. Is there a way to create a robots.txt for this development subdomain only?
Thanks in advance!
-
I would suggest you talk to the developers as Theo suggests to exclude visitors from your test site.
-
The copying is a manual process and I don't want any risks for the live environment. A Httphandler for robots.txt could be a solution and I'm going to discuss this with one of our developers. Other suggestions are still welcome of course!
-
Do you ftp copy one domain to the other? If this is a manual process the excluding the robots.txt that is on the test domain would be as simple as excluding it.
If you automate the copy and want code to function based on base url address then you could create a Httphandler for robots.txt that delivered a different version based on the request url host in the http request header.
-
You could use enviromental variables (for example in your env.ini or config.ini file) that are set to DEVELOPMENT, STAGING, or LIVE based on the appropriate environments the code finds itself in.
With the exact same code, your website would either be limiting IP addresses (on the development environment) or allow all IP addresses (in the live environment). With this setup you can also set different variables per environment such as the level of detail that is shown in your error reporting, connect to a testing database rather than a live one, etc.
[this was supposed to be a reply, but I accidentely clicked the wrong button. Hitting 'Delete reply' results in an error.]
-
Thanks for your quick reply, Theo. Unfortunately, this htpasswd will also get copied to the live environment, so our websites will get password protected live. Could there be any other solution for this?
-
I'm sure there is, but I'm guessing you don't want any human visitors to go to your development subdomain and view what is being done there as well? I'd suggest you either limit the visitors that have access by IP address (thereby effectively blocking out Google in one move) and/or implement a .htpasswd solution where developers can log in with their credentials to your development area (which blocks out Google as well).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Our company creates cobranded subdomains for our clients; does that hurt our SEO?
We create cobranded websites for local businesses in many towns throughout the United States, always under a subdomain of our main site (e.g., afcreditunion.teachbanzai.com). Does this hurt our SEO rankings? We have a very specific reason for creating these microsites, because it's a high selling point. I've read and watched the material here on Moz regarding subdomains and subfolders, but it doesn't quite answer my question: since we create all these microsites not with the intent of passing authority to our website but with the intent of making their microsite have their branding.
Intermediate & Advanced SEO | | teachbanzai0 -
Is Building a Local Directory of Businesses on a Subdomain Good SEO?
Hello Fellow Moz'ers: I own a small digital shop in a major US city. We had a marketing idea which I'd like some input on the soundness of. We are creating a professional services directory of 'digital professional services providers' in our hometown. The directory's membership will only be open to firms located within our city limits. The directory will be curated and maintained, ongoing, by us. Our motivation is 75% selfish and 25% benevolent. The idea is that, by building the directory on our subdomain, we hopefully will collect links, which ultimately will enhance search visibility. But I'm concerned about the devaluation directories have incurred in recent years and I've even seen advice given to the effect that listings in some directories might be harmful to a site's link profile. It is not our intention to harm those who might list in our directory. Any thoughts on this matter would be greatly appreciated!
Intermediate & Advanced SEO | | Daaveey0 -
Robots.txt
Hi all, Happy New Year! I want to block certain pages on our site as they are being flagged (according to my Moz Crawl Report) as duplicate content when in fact that isn't strictly true, it is more to do with the problems faced when using a CMS system... Here are some examples of the pages I want to block and underneath will be what I believe to be the correct robots.txt entry... http://www.XYZ.com/forum/index.php?app=core&module=search&do=viewNewContent&search_app=members&search_app_filters[forums][searchInKey]=&period=today&userMode=&followedItemsOnly= Disallow: /forum/index.php?app=core&module=search http://www.XYZ.com/forum/index.php?app=core&module=reports&rcom=gallery&imageId=980&ctyp=image Disallow: /forum/index.php?app=core&module=reports http://www.XYZ.com/forum/index.php?app=forums&module=post§ion=post&do=reply_post&f=146&t=741&qpid=13308 Disallow: /forum/index.php?app=forums&module=post http://www.XYZ.com/forum/gallery/sizes/182-promenade/small/ http://www.XYZ.com/forum/gallery/sizes/182-promenade/large/ Disallow: /forum/gallery/sizes/ Any help \ advice would be much appreciated. Many thanks Andy
Intermediate & Advanced SEO | | TomKing0 -
When you add 10.000 pages that have no real intention to rank in the SERP, should you: "follow,noindex" or disallow the whole directory through robots? What is your opinion?
I just want a second opinion 🙂 The customer don't want to loose any internal linkvalue by vaporizing link value though a big amount of internal links. What would you do?
Intermediate & Advanced SEO | | Zanox0 -
Subdomain Blog Sitemap link - Add it to regular domain?
Example of setup:
Intermediate & Advanced SEO | | EEE3
www.fancydomain.com
blog.fancydomain.com Because of certain limitations, I'm told we can't put our blogs at the subdirectory level, so we are hosting our blogs at the subdomain level (blog.fancydomain.com). I've been asked to incorporate the blog's sitemap link on the regular domain, or even in the regular domain's sitemap. 1. Putting the a link to blog.fancydomain.com/sitemap_index.xml in the www.fancydomain.com/sitemap.xml -- isn't this against sitemap.org protocol? 2. Is there even a reason to do this? We do have a link to the blog's home page from the www.fancydomain.com navigation, and the blog is set up with its sitemap and link to the sitemap in the footer. 3. What about just including a text link "Blog Sitemap" (linking to blog.fancydomain.com/sitemap_index.html) in the footer of the www.fancydomain.com (adjacent to the text link "Sitemap" which already exists for the www.fancydomain.com's sitemap. Just trying to make sense of this, and figure out why or if it should be done. Thanks!0 -
Folder or subdomain for new e-commerce addition
Our main content site has 5K pieces of unique content all targeting our market. We are planning to add e-commerce as a source monetizing our audience. Should we place the new commerce platform within a subdirectory or subdomain? The layout we are considering is... shop name: Brand Name Market http://www.brandname.com/market/ http://market.brandname.com I am also considering something like: http://www.brandname.com/market/ aggregating product details and content from http://market.brandname.com/ with rel= back to the subdomain if possible.
Intermediate & Advanced SEO | | ejovi0 -
Fourth and Third Level Subdomain Interlinking
Hi everyone. I have a hopefully interesting client question I wanted to pose. I do work for a company with three distinct locations that have unique offerings, service areas, etc. I wouldn't want to see the three locations as subfolders (example company.com/locationone/, company.com/locationtwo/) of one site. They are large and unique presences. Fortunately, they did not organize their locations in such a way, and currently have their locations organized as subdomains, as in locationone.company.com, locationtwo.company.com. I might have preferred locationone.com, locationtwo.com, etc., but that is what I am working with. Their developer has been building new content on fourth level domains, as in newcontent.locationone.company.com and newcontent2.locationone.company.com. In one case one of these fourth level domains also contains a different but parallel checkout process to the one already present on third level domain locationone.company.com. I am looking for advice on how to interlink these sites, and whether to discourage them from building out new fourth level domains (newcontent3.locationone.company.com, etc.) or even to get rid of the current fourth level domains altogether. I'm not sure if the fact that they're subdomains and not subfolders matters as much as it used to. Is this a case of subdomain phobia, or are my concerns justified? Any special advice on dealing with interlinking across fourth, third, and second level domains? Thanks!
Intermediate & Advanced SEO | | rlevers0 -
Partner Login as subdomain?
Hi MozTeam, We have a website that is used as our partner login for our Partners to see their stats, but it is located on a SEPARATE domain from our main corporate website. We currently have thousands of people logging into the external portal every month, which we are obviously not getting good SEO credit for. I am considering bringing the entire login portal into our main corporate website, so that Google sees how popular and useful our site becomes when thousands more people are visiting... We only get a few thousands organic visits to the corporate site per month and about 3x that to the partner login portal. This is why I originally thought we could benefit from bringing it into our corporate site. Challaneges: our website is in .asp but we are launching a new version of it next month, switching it to Wordpress and into .php....but the current partner login website is still in .asp! Questions: 1. How will bringing this site into the main corporate site benefit us as far as SEO? 2. What is the proper way to combine an .asp site with a .php site? 3. If we have to use an iFrame because we can't mix the two languages, will that affect our SEO benefit? Pls advise, as if this is actually a good idea, I'd like to get it launched along with the site redesign that is currently under way.
Intermediate & Advanced SEO | | DerekM880