Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Do I need a separate robots.txt file for my shop subdomain?
-
Hello Mozzers!
Apologies if this question has been asked before, but I couldn't find an answer so here goes...
Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt
We host our shop on a separate subdomain https://shop.mysitename.org.uk
Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!
-
Thank you. I want to disallow specific URLs on the subdomain and add the shop sitemap in the robots.txt file. So I'll go ahead and create another!
-
You go be fine without one. You only need one if you want to manage that subdmain: add specific xml sitemaps links in robots.txt, cut access to specific folders for that subdomain.
if you don't need any of that - just move forward without one.
-
Currently we just have: User-agent: *
I'm in the process of optimising.
-
It depends what currently is in your robots.txt. Usually it would be useful to have another one for your subdomain.
-
Yes, I would have a seperate robots.txt files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving E-Commerce Store to Subdomain?
Hi all, We have a customer who currently uses Square for their in-store point-of-sale system as well as for their e-commerce website. From my understanding, a Square site is a watered-down version of Weebly, and is proving to be highly restrictive from an SEO and content structuring standpoint. It's been an uphill battle to try and get traction for their site in SERPs. Would it be a bad idea to move the entire Square online store to a subdomain, and install WordPress on the root domain? This way their online store would remain as-is, but the primary pages on the site would be on WordPress which would give us a lot more control over the content. I just want to make sure this doesn't negatively impact their SEO. Thanks!
Technical SEO | | suarezventures0 -
Impact of Medium blog hosted on my subdomain
I am using the Medium blogging platform to blog, but it is pointed to my site and appears at blog.mysite.com. Since the content is hosted on Medium and pointed to my subdomain via an A Record / CNAME / etc... 1. Will my domain get credit for backlinks to the blog content? 2. If Medium changes in the future and no longer points to my subdomain, will I lose all of the backlinks I've built up?
Technical SEO | | davidevans_seo0 -
Good robots txt for magento
Dear Communtiy, I am trying to improve the SEO ratings for my website www.rijwielcashencarry.nl (magento). My next step will be implementing robots txt to exclude some crawling pages.
Technical SEO | | rijwielcashencarry040
Does anybody have a good magento robots txt for me? And what need i copy exactly? Thanks everybody! Greetings, Bob0 -
Robots.txt and Multiple Sitemaps
Hello, I have a hopefully simple question but I wanted to ask to get a "second opinion" on what to do in this situation. I am working on a clients robots.txt and we have multiple sitemaps. Using yoast I have my sitemap_index.xml and I also have a sitemap-image.xml I do put them in google and bing by hand but wanted to have it added into the robots.txt for insurance. So my question is, when having multiple sitemaps called out on a robots.txt file does it matter if one is before the other? From my reading it looks like you can have multiple sitemaps called out, but I wasn't sure the best practice when writing it up in the file. Example: User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Sitemap: http://sitename.com/sitemap_index.xml Sitemap: http://sitename.com/sitemap-image.xml Thanks a ton for the feedback, I really appreciate it! :) J
Technical SEO | | allstatetransmission0 -
Will an XML sitemap override a robots.txt
I have a client that has a robots.txt file that is blocking an entire subdomain, entirely by accident. Their original solution, not realizing the robots.txt error, was to submit an xml sitemap to get their pages indexed. I did not think this tactic would work, as the robots.txt would take precedent over the xmls sitemap. But it worked... I have no explanation as to how or why. Does anyone have an answer to this? or any experience with a website that has had a clear Disallow: / for months , that somehow has pages in the index?
Technical SEO | | KCBackofen0 -
My Rankings Keep Going Down - Needs some more ideas on why...
Hi Everyone, I have been working on the website www.PetsSpark.com for a while now. I used to be on the first page for many of my keywords, now I am lucky if i am on the 2nd or 3rd. I stopped my SEO efforts for about 3 months while making changes to the website, its whithin these 3 months that I started dropping from the first page. The biggest thing I notice is the "type" of website now ranking for my main keywords. Mostly Forums, Blogs, or Product Review websites. Take for instance Dog Tear Stain Remover. I used to rank at about #5 for this keyword and now i rank at #20. I'm not sure if my loss of ranking is a combination of things like who Google is letting rank now or if there is something wrong on my website, etc. Can anyone give me a little insight?? Please.... I will also be happy to give more information about what has been going on if needed.
Technical SEO | | DTOSI0 -
Should I set up a disallow in the robots.txt for catalog search results?
When the crawl diagnostics came back for my site its showing around 3,000 pages of duplicate content. Almost all of them are of the catalog search results page. I also did a site search on Google and they have most of the results pages in their index too. I think I should just disallow the bots in the /catalogsearch/ sub folder, but I'm not sure if this will have any negative effect?
Technical SEO | | JordanJudson0
