Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Do I need a separate robots.txt file for my shop subdomain?
-
Hello Mozzers!
Apologies if this question has been asked before, but I couldn't find an answer so here goes...
Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt
We host our shop on a separate subdomain https://shop.mysitename.org.uk
Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!
-
Thank you. I want to disallow specific URLs on the subdomain and add the shop sitemap in the robots.txt file. So I'll go ahead and create another!
-
You go be fine without one. You only need one if you want to manage that subdmain: add specific xml sitemaps links in robots.txt, cut access to specific folders for that subdomain.
if you don't need any of that - just move forward without one.
-
Currently we just have: User-agent: *
I'm in the process of optimising.
-
It depends what currently is in your robots.txt. Usually it would be useful to have another one for your subdomain.
-
Yes, I would have a seperate robots.txt files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt Tester - syntax not understood
I've looked in the robots.txt Tester and I can see 3 warnings: There is a 'syntax not understood' warning for each of these. XML Sitemaps:
Technical SEO | | JamesHancocks1
https://www.pkeducation.co.uk/post-sitemap.xml
https://www.pkeducation.co.uk/sitemap_index.xml How do I fix or reformat these to remove the warnings? Many thanks in advance.
Jim0 -
When should a variant be a variant and when should it be a separate product from an SEO POV?
Hi all, We are looking at changing our current e-commerce store to a new platform and in doing so thinking of making some changes to how we list products in sub-categories. We have seen related questions asking about splitting a single product into multiple products to rank for different terms, but we are wondering about combining multiple products into a single product page? The examples we have seen have been about fashion items with variants of colour and size. However, the products we sell have variances that change the appearance, dimensions and technical specification, so we would like to ask the MOZ community if combining products with these variances would still be deemed good practice? We sell wood burning stoves and a good example of a product that we are considering combining is the Scan 85 stove, which is available in eight different configurations: 85-1, 85-2, 85-3 etc. Scan themselves refer to each version as a separate product and they are bought, stocked and sold as separate products. Wood burning stoves like this typically have a firebox in the centre and then design options that can change the top, side, base, door, colour and fuel. In this example, the firebox is the Scan 85 and the variation is the last number, each of which corresponds to a different design option changing both the appearance and dimensions (see attached image). We have them listed as eight different products on our current site, one for each version. Primarily because each option has its own name (albeit 1-digit difference) which when we created the pages we thought that more pages would present us with more ranking opportunity. However, we have since learnt that because these eight pages are all so similar and it is difficult to write unique content about each product (with the 85-1 and 85-2 the only difference between the models are the black trim on the 85-1 and the silver trim on 85-2). Especially as when talking about the firebox itself, how well the fire burns, how controllable it is etc, will be the same for all versions. Likewise, earning backlinks to eight separate pages is also very difficult. Exploring this lead, us to the question, when is a variant a variant and when is it a separate product? Are there hard and fast rules for what defines variants and products? Or does it simply vary from industry to industry product to product, and if so should we be looking at it from a UX or SEO POV, when making that decision? Our hope is that if we combine these eight products into a single high-quality page, it will present us with a greater ranking opportunity for that one page over eight individual pages. We also hope that in doing so will allow us to create a more intuitive UX on a single page with a unique description, more reviews focused on one page and an explanation of the options available, all of which should lead to more conversions. Finally, by creating a better UX and unique detailed description we hope that there is a higher chance of us earning product level backlinks then we do with eight lower quality pages. One of the issues in creating a single product page for all the variants is the sub-category/results pages, as we would be removing eight simple products and replacing them with one complex product. We have questions over how this would work from a filter/facet level whereby when you apply a filter there is an expectation that the image shown will match the criteria, so if we filter for stoves with a silver trim for example, there is an expectation to only see stoves that have a silver trim in the results. When you have separate product pages you have separate listings which makes this easier to only bring back the models matching the criteria. However, when you have a single page this is more complex as you will need a default image for non-filtered results and then the ability to assign an image to lots of different attributes so that the correct image is always shown that matches the criteria selected. All of which we have been assured is do-able but adds an extra level of complexity to the process from an admin side. The alternative to doing this would be to create eight simple/child products and link them to one configurable/parent product. We could them list the simple products into the results pages and have them all linking back to the main configurable product which could load with the options of the simple product that was selected. From an SEO POV this brings in some more work, redirecting each page to the parent, but ultimately this could provide a better UX and might be the better solution. Has anyone got any experience in doing either of these options before? Both options above with affect the number of products we have available, so does the number of products in a sub-category effect the ability for that category page to rank? We currently have around 500 products in our wood burning stoves category, with perhaps an additional 300 to add. If we go down the combining into a single product page route this will reduce the number of products by around a third. If we keep all the simple/child products, then this will stay around the same. So, have we missed something obvious? Is there a glaring issue that we have overlooked from an SEO point of view as well as from the customer experience? We would appreciate your thoughts on this. Thanks, Reece scan85-1.jpg
Technical SEO | | fireproductsuk0 -
One robots.txt file for multiple sites?
I have 2 sites hosted with Blue Host and was told to put the robots.txt in the root folder and just use the one robots.txt for both sites. Is this right? It seems wrong. I want to block certain things on one site. Thanks for the help, Rena
Technical SEO | | renalynd270 -
Should I block Map pages with robots.txt?
Hello, I have a website that was started in 1999. On the website I have map pages for each of the offices listed on my site, for which there are about 120. Each of the 120 maps is in a whole separate html page. There is no content in the page other than the map. I know all of the offices love having the map pages so I don't want to remove the pages. So, my question is would these pages with no real content be hurting the rankings of the other pages on our site? Therefore, should I block the pages with my robots.txt? Would I also have to remove these pages (in webmaster tools?) from Google for blocking by robots.txt to really work? I appreciate your feedback, thanks!
Technical SEO | | imaginex0 -
Adding multi-language sitemaps to robots.txt
I am working on a revamped multi-language site that has moved to Magento. Each language runs off the core coding so there are no sub-directories per language. The developer has created sitemaps which have been uploaded to their respective GWT accounts. They have placed the sitemaps in new directories such as: /sitemap/uk/sitemap.xml /sitemap/de/sitemap.xml I want to add the sitemaps to the robots.txt but can't figure out how to do it. Also should they have placed the sitemaps in a single location with the file identifying each language: /sitemap/uk-sitemap.xml /sitemap/de-sitemap.xml What is the cleanest way of handling these sitemaps and can/should I get them on robots.txt?
Technical SEO | | MickEdwards0 -
Do I need an XML sitemap?
I have an established website that ranks well in Google. However, I have just noticed that no xml sitemap has been registered in Google webmaster tools, so the likelihood is that it hasn't been registered with the other search engines. However, there is an html sitemap listed on the website. Seeing as the website is already ranking well, do I still need to generate and submit an XML sitemap? Could there be any detriment to current rankings in doing so?
Technical SEO | | pugh0 -
Internal search : rel=canonical vs noindex vs robots.txt
Hi everyone, I have a website with a lot of internal search results pages indexed. I'm not asking if they should be indexed or not, I know they should not according to Google's guidelines. And they make a bunch of duplicated pages so I want to solve this problem. The thing is, if I noindex them, the site is gonna lose a non-negligible chunk of traffic : nearly 13% according to google analytics !!! I thought of blocking them in robots.txt. This solution would not keep them out of the index. But the pages appearing in GG SERPS would then look empty (no title, no description), thus their CTR would plummet and I would lose a bit of traffic too... The last idea I had was to use a rel=canonical tag pointing to the original search page (that is empty, without results), but it would probably have the same effect as noindexing them, wouldn't it ? (never tried so I'm not sure of this) Of course I did some research on the subject, but each of my finding recommanded one of the 3 methods only ! One even recommanded noindex+robots.txt block which is stupid because the noindex would then be useless... Is there somebody who can tell me which option is the best to keep this traffic ? Thanks a million
Technical SEO | | JohannCR0 -
Robots.txt and canonical tag
In the SEOmoz post - http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts, it's being said - If you have a robots.txt disallow in place for a page, the canonical tag will never be seen. Does it so happen that if a page is disallowed by robots.txt, spiders DO NOT read the html code ?
Technical SEO | | seoug_20050