Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Do I need a separate robots.txt file for my shop subdomain?
-
Hello Mozzers!
Apologies if this question has been asked before, but I couldn't find an answer so here goes...
Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt
We host our shop on a separate subdomain https://shop.mysitename.org.uk
Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!
-
Thank you. I want to disallow specific URLs on the subdomain and add the shop sitemap in the robots.txt file. So I'll go ahead and create another!
-
You go be fine without one. You only need one if you want to manage that subdmain: add specific xml sitemaps links in robots.txt, cut access to specific folders for that subdomain.
if you don't need any of that - just move forward without one.
-
Currently we just have: User-agent: *
I'm in the process of optimising.
-
It depends what currently is in your robots.txt. Usually it would be useful to have another one for your subdomain.
-
Yes, I would have a seperate robots.txt files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl solutions for landing pages that don't contain a robots.txt file?
My site (www.nomader.com) is currently built on Instapage, which does not offer the ability to add a robots.txt file. I plan to migrate to a Shopify site in the coming months, but for now the Instapage site is my primary website. In the interim, would you suggest that I manually request a Google crawl through the search console tool? If so, how often? Any other suggestions for countering this Meta Noindex issue?
Technical SEO | | Nomader1 -
Reviews on Product Page or Separated
Good Afternoon We currently have our individual product information pages set-up with a link through to a separate review page optimised for the term "Product A Reviews" I was reading about structured data and if I read correctly, the reviews should sit with the marked up product data so I was wondering whether to merge them back into one page. We have many reviews so the review pages are paginated in blocks of 25 My options are: Leave as it is, product info page and separate review page Merge the review content back in to the main page and have the pagination work on that page Include the first 25 reviews on the product info page then when user clicks through to page 2, 3 etc they're taken to the separated review page. In that way the product page would regularly get new content and we can still have a page specifically targeted for reviews. From the users point of view, they probably aren't even aware they're being taken to a separate reviews page so with that in mind as I'm typing this maybe they should be one page again
Technical SEO | | Ham19790 -
Is it important to include image files in your sitemap?
I run an ecommerce business that has over 4000 product pages which, as you can imagine, branches off into thousands of image files. Is it necessary to include those in my sitemap for faster indexing? Thanks for you help! -Reed
Technical SEO | | IceIcebaby0 -
My Rankings Keep Going Down - Needs some more ideas on why...
Hi Everyone, I have been working on the website www.PetsSpark.com for a while now. I used to be on the first page for many of my keywords, now I am lucky if i am on the 2nd or 3rd. I stopped my SEO efforts for about 3 months while making changes to the website, its whithin these 3 months that I started dropping from the first page. The biggest thing I notice is the "type" of website now ranking for my main keywords. Mostly Forums, Blogs, or Product Review websites. Take for instance Dog Tear Stain Remover. I used to rank at about #5 for this keyword and now i rank at #20. I'm not sure if my loss of ranking is a combination of things like who Google is letting rank now or if there is something wrong on my website, etc. Can anyone give me a little insight?? Please.... I will also be happy to give more information about what has been going on if needed.
Technical SEO | | DTOSI0 -
301 Redirect on a PDF, DOCX files?
Hi, I have to rename many pdf and docx files. How can I implement 301 redirect on them as they are linked from 'n' number of places? Regards, Shailendra Sial
Technical SEO | | IM_Learner1 -
Do I need an XML sitemap?
I have an established website that ranks well in Google. However, I have just noticed that no xml sitemap has been registered in Google webmaster tools, so the likelihood is that it hasn't been registered with the other search engines. However, there is an html sitemap listed on the website. Seeing as the website is already ranking well, do I still need to generate and submit an XML sitemap? Could there be any detriment to current rankings in doing so?
Technical SEO | | pugh0 -
Microsite on subdomain vs. subdirectory
Based on this post from 2009, it's recommended in most situations to set up a microsite as a subdirectory as opposed to a subdomain. http://www.seomoz.org/blog/understanding-root-domains-subdomains-vs-subfolders-microsites. The primary argument seems to be that the search engines view the subdomain as a separate entity from the domain and therefore, the subdomain doesn't benefit from any of the trust rank, quality scores, etc. Rand made a comment that seemed like the subdomain could SOMETIMES inherit some of these factors, but didn't expound on those instances. What determines whether the search engine will view your subdomain hosted microsite as part of the main domain vs. a completely separate site? I read it has to do with the interlinking between the two.
Technical SEO | | ryanwats0 -
How to move my blog from subdomain to subfolder?
Not an unusual situation, I have a blog on blog.domain.com it has quite a few blog postings. The platform is old and will be scrapped, but the blog content itself is going to be moved to domain.com/blog. The current process is we are manually listing all linked to/content pages and we are going to 301 redirect them to their counterparts on the new blog. This is going to be a tedious process. A) Is there any way to automate the moving of the blog? B) What is the best way to do the massive 301 redirect, php headers, .htaccess? Should we move the individual pages with redirects, or redirect the domain in the .htaccess (this will be very difficult to match all the titles and file structure)?
Technical SEO | | MarloSchneider0