Block an entire subdomain with robots.txt?
-
Is it possible to block an entire subdomain with robots.txt?
I write for a blog that has their root domain as well as a subdomain pointing to the exact same IP. Getting rid of the option is not an option so I'd like to explore other options to avoid duplicate content. Any ideas?
-
Awesome! That did the trick -- thanks for your help. The site is no longer listed
-
Fact is, the robots file alone will never work (the link has a good explanation why - short form: all it does is stop the bots from indexing again).
Best to request removal then wait a few days.
-
Yeah. As of yet, the site has not been de-indexed. We placed the conditional rule in htaccess and are getting different robots.txt files for the domain and subdomain -- so that works. But I've never done this before so I don't know how long it's supposed to take?
I'll try to verify via Webmaster Tools to speed up the process. Thanks
-
You should do a remove request in Google Webmaster Tools. You have to first verify the sub-domain then request the removal.
See this post on why the robots file alone won't work...
http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts
-
Awesome. We used your second idea and so far it looks like it is working exactly how we want. Thanks for the idea.
Will report back to confirm that the subdomain has been de-indexed.
-
Option 1 could come with a small performance hit if you have a lot of txt files being used on the server.
There shouldn't be any negative side effects to option 2 if the rewrite is clean (IE not accidently a redirect) and the content of the two files are robots compliant.
Good luck
-
Thanks for the suggestion. I'll definitely have to do a bit more research into this one to make sure that it doesn't have any negative side effects before implementation
-
We have a plugin right now that places canonical tags, but unfortunately, the canonical for the subdomain points to the subdomain. I'll look around to see if I can tweak the settings
-
Sounds like (from other discussions) you may be stuck requiring a dynamic robot.txt file which detects what domain the bot is on and changes the content accordingly. This means the server has to run all .txt file as (I presume) PHP.
Or, you could conditionally rewrite the /robot.txt URL to a new file according to sub-domain
RewriteEngine on
RewriteCond %{HTTP_HOST} ^subdomain.website.com$
RewriteRule ^robotx.txt$ robots-subdomain.txtThen add:
User-agent: *
Disallow: /to the robots-subdomain.txt file
(untested)
-
Placing canonical tags isn't an option? Detect that the page is being viewed through the subdomain, and if so, write the canonical tag on the page back to the root domain?
Or, just place a canonical tag on every page pointing back to the root domain (so the subdomain and root domain pages would both have them). Apparently, it's ok to have a canonical tag on a page pointing to itself. I haven't tried this, but if Matt Cutts says it's ok...
-
Hey Ryan,
I wasn't directly involved with the decision to create the subdomain, but I'm told that it is necessary to create in order to bypass certain elements that were affecting the root domain.
Nevertheless, it is a blog and the users now need to login to the subdomain in order to access the Wordpress backend to bypass those elements. Traffic for the site still goes to the root domain.
-
They both point to the same location on the server? So there's not a different folder for the subdomain?
If that's the case then I suggest adding a rule to your htaccess file to 301 the subdomain back to the main domain in exactly the same way people redirect from non-www to www or vice-versa. However, you should ask why the server is configured to have a duplicate subdomain? You might just edit your apache settings to get rid of that subdomain (usually done through a cpanel interface).
Here is what your htaccess might look like:
<ifmodule mod_rewrite.c="">RewriteEngine on
# Redirect non-www to wwww
RewriteCond %{HTTP_HOST} !^www.mydomain.org [NC]
RewriteRule ^(.*)$ http://www.mydomain.org/$1 [R=301,L]</ifmodule> -
Not to me LOL I think you'll need someone with a bit more expertise in this area than I to assist in this case. Kyle, I'm sorry I couldn't offer more assistance... but I don't want to tell you something if I'm not 100% sure. I suspect one of the many bright SEOmozer's will quickly come to the rescue on this one.
Andy
-
Hey Andy,
Herein lies the problem. Since the domain and subdomain point to the exact same place, they both utilize the same robots.txt file.
Does that make sense?
-
Hi Kyle Yes, you can block an entire subdomain via robots.txt, however you'll need to create a robots.txt file and place it in the root of the subdomain, then add the code to direct the bots to stay away from the entire subdomain's content.
User-agent: *
Disallow: /hope this helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disallow URLs ENDING with certain values in robots.txt?
Is there any way to disallow URLs ending in a certain value? For example, if I have the following product page URL: http://website.com/category/product1, and I want to disallow /category/product1/review, /category/product2/review, etc. without disallowing the product pages themselves, is there any shortcut to do this, or must I disallow each gallery page individually?
Intermediate & Advanced SEO | | jmorehouse0 -
Blocking some countries and redirecting that traffic
Hi there, I have a video site, which is on CDN and is really expensive to run. So I want to block most of the countries and only keep HQ ones. I wonder if there's a difference if I just block them and show blank page, or if I show them a page with text and let's say a link to a different site or if I just simply redirect to some other site. Do you think I can still get good ranking on google on countries that I don't block?
Intermediate & Advanced SEO | | melbog0 -
301 redirect subdomain to path and 301 for popular pages
We have very popular pages that have many backlinks. www.chezmaya.com/jeux/game33.htm have so many backlinks and it's very popular. Now If i'm moving this page to a new path like : http://www.chezmaya.com/jeux/component/mtree/Défouloir/Game33/details.html with a 301. Your SEOmoz toolbar is now giving a very low PA:1 and mR:0.00 for this new page. My question is after you crawl my site again would you change the values to what /jeux/game33.htm got before ? We used to have jeux.chezmaya.com and moved to www.chezmaya.com/jeux/
Intermediate & Advanced SEO | | SocialGeekMedia
Same here PA:1 and mR:0.00 for this page. Also Matt Cutts say that Google does transfer the juice from the old page to the new one. I already saw one url changed in a search for puzzle, it's at the same position it was before, but it say's 6 days ago beside. So I wonder if this is temporary and it will move with time? Thanks0 -
Domain vs Subdomain for Multi-Location Practice
I have a client who has 2 locations (Orlando & Tampa) and would like to keep the current domain for both locations (DA 29). We want to target additional cities within each service area (Orlando & Tampa). Each service area would target 2 cities on the main pages and 4-5 cities with "SEO" pages which contains unique content specific to the given city. Would I be better off creating sub domains (www.orlando.domain.com & www.tampa.domain.com), creating subfolders (www.domain.com/orlando, etc) or keeping the domain as is and create SEO pages specific to each city? We want to spread the domain authority to both locations.
Intermediate & Advanced SEO | | Red_Spot_Interactive0 -
The use of subdomains to improve SEO?
A clients website which provide a number of trade services which have a page for each service they provide for example: carpentry or electrician or plumbing etc. currently these pages are found at domain.co.uk/bathrooms/ bathrooms.html I am trying to optmise each page better as they are competing with other sites who for example sell bathrooms rather than bathroom installers or plumbers. As part of the on page optimisation I plan to change the page names and directory structure. I had an idea to split the website down into subdomains for various sections i.e for all their services Create a sub domain such as http://plumber.domain.co.uk 2.) upload the relevant content (in this example the plumbing page) to the sub domain location 3.) correct all the links to absolute URLs for each sub domain / Will this help target better use of keywords in the URL in terms of SEO efforts ? hope it makes sense thanks Darren
Intermediate & Advanced SEO | | Bristolweb0 -
Robots.txt is blocking Wordpress Pages from Googlebot?
I have a robots.txt file on my server, which I did not develop, it was done by the web designer at the company before me. Then there is a word press plugin that generates a robots.txt file. How Do I unblock all the wordpress pages from googlebot?
Intermediate & Advanced SEO | | ENSO0 -
XML Sitemap instruction in robots.txt = Worth doing?
Hi fellow SEO's, Just a quick one, I was reading a few guides on Bing Webmaster tools and found that you can use the robots.txt file to point crawlers/bots to your XML sitemap (they don't look for it by default). I was just wondering if it would be worth creating a robots.txt file purely for the purpose of pointing bots to the XML sitemap? I've submitted it manually to Google and Bing webmaster tools but I was thinking more for the other bots (I.e. Mozbot, the SEOmoz bot?). Any thoughts would be appreciated! 🙂 Regards, Ash
Intermediate & Advanced SEO | | AshSEO20110 -
What are the Best Practices for moving a blog from subdomain to domain/subcategory?
Howdy SEOmoz fans! (couldn't resist). I'm moving a wordpress blog from blog.domain.com to domain.com/blog. Trying to do it right the first time and cover all my bases. Issues I'm trying to handle correctly, in varying degrees of importance: External LInks Internal Links Google Friendly Traffic Routing in a dynamic environment (wordpress, 301, .htaccess, etc.) Thanks so much for any and all input!
Intermediate & Advanced SEO | | NTM1