Subdomain Robots.txt
-
If I have a subdomain (a blog) that is having tags and categories indexed when they should not be, because they are creating duplicate content. Can I block them using a robots.txt file? Can I/do I need to have a separate robots file for my subdomain?
If so, how would I format it? Do I need to specify that it is a subdomain robots file, or will the search engines automatically pick this up?
Thanks!
-
Thanks Wissam. I was thinking this was the way to go, and I appreciate your input.
I do use the Yoast SEO plugin for Wordpress on another site, but the blog in question is through BlogEngine. I will do what you have suggested.
Cheers!
-
if the url is http://blog.website.com
then the Robots.txt should be accessable threw http://blog.website.com/robots.txt
I would suggest these steps
- Verify your blog the Google webmaster tools
- generate a robots .txt file with Google webmaster tools
- Upload it to the Subdomain.
There is another way if you are using Wordpress.
There is a All in One SEO plugin / Wordpress SEO by Yoast. threw the settings you can specify to add NOINDEX to all Category, tags, author and others. its faster and error free.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No index tag robots.txt
Hi Mozzers, A client's website has a lot of internal directories defined as /node/*. I already added the rule 'Disallow: /node/*' to the robots.txt file to prevents bots from crawling these pages. However, the pages are already indexed and appear in the search results. In an article of Deepcrawl, they say you can simply add the rule 'Noindex: /node/*' to the robots.txt file, but other sources claim the only way is to add a noindex directive in the meta robots tag of every page. Can someone tell me which is the best way to prevent these pages from getting indexed? Small note: there are more than 100 pages. Thanks!
Technical SEO | | WeAreDigital_BE
Jens0 -
Should a login page for a payroll / timekeeping comp[any be no follow for robots.txt?
I am managing a Timekeeping/Payroll company. My question is about the customer login page. Would this typically be nofollow for robots?
Technical SEO | | donsilvernail0 -
Robots.txt
I have a client who after designer added a robots.txt file has experience continual growth of urls blocked by robots,tx but now urls blocked (1700 aprox urls) has surpassed those indexed (1000). Surely that would mean all current urls are blocked (plus some extra mysterious ones). However pages still listing in Google and traffic being generated from organic search so doesnt look like this is the case apart from the rather alarming webmaster tools report any ideas whats going on here ? cheers dan
Technical SEO | | Dan-Lawrence0 -
Blocked by robots
my client GWT has a number of notices for "blocked by meta-robots" - these are all either blog posts/categories/or tags his former seo told him this: "We've activated following settings: Use noindex for Categories Use noindex for Archives Use noindex for Tag Archives to reduce keyword stuffing & duplicate post tags
Technical SEO | | Ezpro9
Disabling all 3 noindex settings above may remove google blocks but also will send too many similar tags, post archives/category. " is this guy correct? what would be the problem with indexing these? am i correct in thinking they should be indexed? thanks0 -
Bandcamp subdomain
I have a website - www.weddingmusicproject.com that is doing quite well. However, we have all of the actual music listed on weddingmusicproject.bandcamp.com and have a number of very powerful pages there (bandcamp is a great product by the way). Is there a strong benefit to our domain's authority if we move our "bandcamp" site onto our own subdomain using their custom domain option - http://bandcamp.com/faq_custom_domains - Something like music.weddingmusicproject.com. I seem to think that it would increase our overall domain authority, but it wouldn't increase the number of inbound links or anything. If anything it would decrease the number of linking domains and bandcamp is quite a powerful site so those links would just turn into internal links. Thoughts? I know this is probably a basic concept, but I've thought it over a number of times and can't come to a conclusion.
Technical SEO | | deyobr0 -
Can I Disallow Faceted Nav URLs - Robots.txt
I have been disallowing /*? So I know that works without affecting crawling. I am wondering if I can disallow the faceted nav urls. So disallow: /category.html/? /category2.html/? /category3.html/*? To prevent the price faceted url from being cached: /category.html?price=1%2C1000
Technical SEO | | tylerfraser
and
/category.html?price=1%2C1000&product_material=88 Thanks!0 -
Moving subdomain ? How to ?
Hi all. I've purchased a domain name two years ago with the idea to offer wide range of services. I've also created a sub-domain providing specific service for highly competitive keyword. Sadly plans went wrong and I didn't use the root domain name at all, just the sub-domain providing that service. There aren't much links to that sub-domain, but all are quality links, until recently I've managed to keep positions between 5 and 7 without any effort, but yesterday I saw that it's dropped to 9. The question is, before I start to build links and write articles to get back up my domain, is it worth to move that sub-domain to my original root domain. As I said, there aren't much links to that sub-domain, it only has pagerank1, also for the last year the original root domain was redirected (301) to the sub-domain to not loose traffic and I'm scared if I reverse this procedure and redirect my sub-domain to the root domain that Google will get confused. It's a tricky question I know 🙂
Technical SEO | | VasilTasev0 -
What are the SEO related negative aspects of having a blog on a subdomain?
Just double checking this one. Is it less than ideal to have a blog on a subdomain rather than as a sub folder on a domain?
Technical SEO | | PerchDigital0