Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Do I need a separate robots.txt file for my shop subdomain?
-
Hello Mozzers!
Apologies if this question has been asked before, but I couldn't find an answer so here goes...
Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt
We host our shop on a separate subdomain https://shop.mysitename.org.uk
Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!
-
Thank you. I want to disallow specific URLs on the subdomain and add the shop sitemap in the robots.txt file. So I'll go ahead and create another!
-
You go be fine without one. You only need one if you want to manage that subdmain: add specific xml sitemaps links in robots.txt, cut access to specific folders for that subdomain.
if you don't need any of that - just move forward without one.
-
Currently we just have: User-agent: *
I'm in the process of optimising.
-
It depends what currently is in your robots.txt. Usually it would be useful to have another one for your subdomain.
-
Yes, I would have a seperate robots.txt files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When should a variant be a variant and when should it be a separate product from an SEO POV?
Hi all, We are looking at changing our current e-commerce store to a new platform and in doing so thinking of making some changes to how we list products in sub-categories. We have seen related questions asking about splitting a single product into multiple products to rank for different terms, but we are wondering about combining multiple products into a single product page? The examples we have seen have been about fashion items with variants of colour and size. However, the products we sell have variances that change the appearance, dimensions and technical specification, so we would like to ask the MOZ community if combining products with these variances would still be deemed good practice? We sell wood burning stoves and a good example of a product that we are considering combining is the Scan 85 stove, which is available in eight different configurations: 85-1, 85-2, 85-3 etc. Scan themselves refer to each version as a separate product and they are bought, stocked and sold as separate products. Wood burning stoves like this typically have a firebox in the centre and then design options that can change the top, side, base, door, colour and fuel. In this example, the firebox is the Scan 85 and the variation is the last number, each of which corresponds to a different design option changing both the appearance and dimensions (see attached image). We have them listed as eight different products on our current site, one for each version. Primarily because each option has its own name (albeit 1-digit difference) which when we created the pages we thought that more pages would present us with more ranking opportunity. However, we have since learnt that because these eight pages are all so similar and it is difficult to write unique content about each product (with the 85-1 and 85-2 the only difference between the models are the black trim on the 85-1 and the silver trim on 85-2). Especially as when talking about the firebox itself, how well the fire burns, how controllable it is etc, will be the same for all versions. Likewise, earning backlinks to eight separate pages is also very difficult. Exploring this lead, us to the question, when is a variant a variant and when is it a separate product? Are there hard and fast rules for what defines variants and products? Or does it simply vary from industry to industry product to product, and if so should we be looking at it from a UX or SEO POV, when making that decision? Our hope is that if we combine these eight products into a single high-quality page, it will present us with a greater ranking opportunity for that one page over eight individual pages. We also hope that in doing so will allow us to create a more intuitive UX on a single page with a unique description, more reviews focused on one page and an explanation of the options available, all of which should lead to more conversions. Finally, by creating a better UX and unique detailed description we hope that there is a higher chance of us earning product level backlinks then we do with eight lower quality pages. One of the issues in creating a single product page for all the variants is the sub-category/results pages, as we would be removing eight simple products and replacing them with one complex product. We have questions over how this would work from a filter/facet level whereby when you apply a filter there is an expectation that the image shown will match the criteria, so if we filter for stoves with a silver trim for example, there is an expectation to only see stoves that have a silver trim in the results. When you have separate product pages you have separate listings which makes this easier to only bring back the models matching the criteria. However, when you have a single page this is more complex as you will need a default image for non-filtered results and then the ability to assign an image to lots of different attributes so that the correct image is always shown that matches the criteria selected. All of which we have been assured is do-able but adds an extra level of complexity to the process from an admin side. The alternative to doing this would be to create eight simple/child products and link them to one configurable/parent product. We could them list the simple products into the results pages and have them all linking back to the main configurable product which could load with the options of the simple product that was selected. From an SEO POV this brings in some more work, redirecting each page to the parent, but ultimately this could provide a better UX and might be the better solution. Has anyone got any experience in doing either of these options before? Both options above with affect the number of products we have available, so does the number of products in a sub-category effect the ability for that category page to rank? We currently have around 500 products in our wood burning stoves category, with perhaps an additional 300 to add. If we go down the combining into a single product page route this will reduce the number of products by around a third. If we keep all the simple/child products, then this will stay around the same. So, have we missed something obvious? Is there a glaring issue that we have overlooked from an SEO point of view as well as from the customer experience? We would appreciate your thoughts on this. Thanks, Reece scan85-1.jpg
Technical SEO | | fireproductsuk0 -
Robots.txt & meta noindex--site still shows up on Google Search
I have set up my robots.txt like this: User-agent: *
Technical SEO | | RoxBrock
Disallow: / and I have this meta tag in my on a Wordpress site, set up with SEO Yoast name="robots" content="noindex,follow"/> I did "Fetch as Google" on my Google Search Console My website is still showing up in the search results and it says this: "A description for this result is not available because of this site's robots.txt" This site has not shown up for years and now it is ranking above my site that I want to rank for this keyword. How do I get Google to ignore this site? This seems really weird and I'm confused how a site with little content, that has not been updated for years can rank higher than a site that is constantly updated and improved.1 -
Is sitemap required on my robots.txt?
Hi, I know that linking your sitemap from your robots.txt file is a good practice. Ok, but... may I just send my sitemap to search console and forget about adding ti to my robots.txt? That's my situation: 1 multilang platform which means... ... 2 set of pages. One for each lang, of course But my CMS (magento) only allows me to have 1 robots.txt file So, again: may I have a robots.txt file woth no sitemap AND not suffering any potential SEO loss? Thanks in advance, Juan Vicente Mañanas Abad
Technical SEO | | Webicultors0 -
Robots.txt on subdomains
Hi guys! I keep reading conflicting information on this and it's left me a little unsure. Am I right in thinking that a website with a subdomain of shop.sitetitle.com will share the same robots.txt file as the root domain?
Technical SEO | | Whittie0 -
Robots.txt on http vs. https
We recently changed our domain from http to https. When a user enters any URL on http, there is an global 301 redirect to the same page on https. I cannot find instructions about what to do with robots.txt. Now that https is the canonical version, should I block the http-Version with robots.txt? Strangely, I cannot find a single ressource about this...
Technical SEO | | zeepartner0 -
No index on subdomains
Hi, We have a subdomain that is appearing in the search results - I want to hide this as it looks really bad. If I were to add the no index tag to the sub domain would URL would this affect the whole domain or just that sub domain? The main domain is vitally important - it is just that sub domain I need to hide. Many thanks
Technical SEO | | Creditsafe0 -
Block Domain in robots.txt
Hi. We had some URLs that were indexed in Google from a www1-subdomain. We have now disabled the URLs (returning a 404 - for other reasons we cannot do a redirect from www1 to www) and blocked via robots.txt. But the amount of indexed pages keeps increasing (for 2 weeks now). Unfortunately, I cannot install Webmaster Tools for this subdomain to tell Google to back off... Any ideas why this could be and whether it's normal? I can send you more domain infos by personal message if you want to have a look at it.
Technical SEO | | zeepartner0 -
Exact match subdomains
Hi, I have seen significant SEO benefits from owning exact match domains and was wondering whether exact match subdomains hold the same (or some) of these benefits? eg. halloweencostumes.co.uk vs. halloween [dot] costumes.co.uk Many thanks.
Technical SEO | | martyc0