Robots.txt on subdomains
-
Hi guys!
I keep reading conflicting information on this and it's left me a little unsure. Am I right in thinking that a website with a subdomain of shop.sitetitle.com will share the same robots.txt file as the root domain?
-
That's about as comprehensive an answer as I could have hoped for. Thanks Ryan, really appreciated.
-
Mostly no. I say 'mostly' because a lot of times when you look at a site using www and no-www if both of those work they're almost always pulling files from the same location (hence the warnings around duplicate content), so both www.domain.com/robots.txt and domain.com/robots.txt are going to work. This is the dominant example of a subdomain sharing a robots.txt file. However, on domains that are set up as their own subdomains they have different robots.txt. Take a look at the many differences between subdomain1-1000.wordpress.com/robots.txt vs wordpress.com/robots.txt. If you set up a subdomain that isn't just a reflection of your root domain, then you'll need to create a robots.txt file as well. Cheers!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blocking pages from Moz and Alexa robots
Hello, We want to block all pages in this directory from Moz and Alexa robots - /slabinventory/search/ Here is an example page - https://www.msisurfaces.com/slabinventory/search/granite/giallo-fiesta/los-angeles-slabs/msi/ Let me know if this is a valid disallow for what I'm trying to. User-agent: ia_archiver
Technical SEO | | Pushm
Disallow: /slabinventory/search/* User-agent: rogerbot
Disallow: /slabinventory/search/* Thanks.0 -
Should a login page for a payroll / timekeeping comp[any be no follow for robots.txt?
I am managing a Timekeeping/Payroll company. My question is about the customer login page. Would this typically be nofollow for robots?
Technical SEO | | donsilvernail0 -
Keyword Suggestions Tool & Different Subdomains
Hey all, Was reading Dan Shure's brilliant post on the Keyword Planner, and decided to plug a few of my own pages into the URL-suggester tool as well. What I got back was nothing short of strange. After plugging in our Features page, (which describes our Social Media Contesting Platform,) and getting back a bunch of suggestions related to Dr Seuss and Interior Design Scholarships, I realized that the Keyword Suggestion tool was being broken by our subdomains. I looked for precedent on my particular issue, but I think I might not be searching properly. Could anyone provide any insight into whether or not this might affect how spiders see the content on Strutta.com, whether or not this is just something that will affect the Keyword Suggestions Tool or actual SERP rankings, and if this content is already present elsewhere on MOZ, a link to said content? Much obliged 🙂
Technical SEO | | danny.wood0 -
Adding multi-language sitemaps to robots.txt
I am working on a revamped multi-language site that has moved to Magento. Each language runs off the core coding so there are no sub-directories per language. The developer has created sitemaps which have been uploaded to their respective GWT accounts. They have placed the sitemaps in new directories such as: /sitemap/uk/sitemap.xml /sitemap/de/sitemap.xml I want to add the sitemaps to the robots.txt but can't figure out how to do it. Also should they have placed the sitemaps in a single location with the file identifying each language: /sitemap/uk-sitemap.xml /sitemap/de-sitemap.xml What is the cleanest way of handling these sitemaps and can/should I get them on robots.txt?
Technical SEO | | MickEdwards0 -
Do knowledge base plugins on a subdomain have seo benefit?
Hi SEO Moz, We want to use a knowledge base plugin (cheaper & faster than building it ourselves) where we can have Q&As for our website. We want this to help with our SEO by adding in our keyterms that we want to rank for. We've looked into TenderApp & Get Satisfaction which look like good solutions - however, as they're both on a sub-domain do we get any seo benefit from this? When people link to our knowledge base, will this help our website at all - or is the benefit going to go to TenderApp/Get Satisfaction? For instance - Our website URL is http://widget.products.com The KB URL is http://widgetsupport.products.com If the above plugin is not a good solution, is there anything else that is better? Any help will be greatly appreciated!! Thanks.
Technical SEO | | qdigi0 -
Getting subdomains unindexed
If i turn an application off displaying a 503 error will that get my site unindexed from search engines?
Technical SEO | | kchandler0 -
Will errors on a subdomain effect the overall health of the root domain?
As stated in the question, we have 2 sub domains that contain over 2000 reported errors from SEOMOZ. The root domain has a clean bill of health, and i was just wondering if these errors on the sub-domains could have a negative effect on the root domain in the eyes of Google. Your comments will be appreciated. Regards Greg
Technical SEO | | AndreVanKets0 -
Robots.txt question
I want to block spiders from specific specific part of website (say abc folder). In robots.txt, i have to write - User-agent: * Disallow: /abc/ Shall i have to insert the last slash. or will this do User-agent: * Disallow: /abc
Technical SEO | | seoug_20050
