Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Do I need a separate robots.txt file for my shop subdomain?
-
Hello Mozzers!
Apologies if this question has been asked before, but I couldn't find an answer so here goes...
Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt
We host our shop on a separate subdomain https://shop.mysitename.org.uk
Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!
-
Thank you. I want to disallow specific URLs on the subdomain and add the shop sitemap in the robots.txt file. So I'll go ahead and create another!
-
You go be fine without one. You only need one if you want to manage that subdmain: add specific xml sitemaps links in robots.txt, cut access to specific folders for that subdomain.
if you don't need any of that - just move forward without one.
-
Currently we just have: User-agent: *
I'm in the process of optimising.
-
It depends what currently is in your robots.txt. Usually it would be useful to have another one for your subdomain.
-
Yes, I would have a seperate robots.txt files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Role of Robots.txt and Search Console parameters settings
Hi, wondering if anyone can point me to resources or explain the difference between these two. If a site has url parameters disallowed in Robots.txt is it redundant to edit settings in Search Console parameters to anything other than "Let Googlebot Decide"?
Technical SEO | | LivDetrick0 -
Crawl solutions for landing pages that don't contain a robots.txt file?
My site (www.nomader.com) is currently built on Instapage, which does not offer the ability to add a robots.txt file. I plan to migrate to a Shopify site in the coming months, but for now the Instapage site is my primary website. In the interim, would you suggest that I manually request a Google crawl through the search console tool? If so, how often? Any other suggestions for countering this Meta Noindex issue?
Technical SEO | | Nomader1 -
Robots txt. in page with 301 redirect
We currently have a a series of help pages that we would like to disallow from our robots txt. The thing is that these help pages are located in our old website, which now has a 301 redirect to current site. Which is the proper way to go around? 1- Add the pages we want to disallow to the robots.txt of the new website? 2- Break the redirect momentarily and add the pages to the robots.txt of the old one? Thanks
Technical SEO | | Kilgray0 -
Good robots txt for magento
Dear Communtiy, I am trying to improve the SEO ratings for my website www.rijwielcashencarry.nl (magento). My next step will be implementing robots txt to exclude some crawling pages.
Technical SEO | | rijwielcashencarry040
Does anybody have a good magento robots txt for me? And what need i copy exactly? Thanks everybody! Greetings, Bob0 -
Does an subdomain hosted offsite provide SEO value
We have a job board hosted through an applicant processing system which we've setup as a subdomain (jobs.ourcompany.com), most of the assets are hosted on our primary domain (ourcompany.com). My question is does having it hosted offsite provide any value? Do we get credit for that content being shared and distributed on the web or does the applicant processing system? As I see it the options are (correct me if I'm wrong): Host the job listings on our primary domain (ourcompany.com/jobs) and have it point to the application on the subdomain. Advertise the job listings pointing to the primary domain on the paid sites. The free job listing sites will automatically point to the sub-domain because the applicant processing system automatically submits them. Host the job listings entirely on the sub-domain applicant tracking system and link to it from our primary site navigation. Advertise the job listings to the sub-domain so that both free and paid point to the same place. Obviously the second one would be much easier just not sure on the technical side of our website getting credit by search engines as the one who has produced the content.
Technical SEO | | r1200gsa0 -
Shopping Carts & Sub Domains
I was hoping someone could guide me in making the correct decision regarding integrating my existing domain with a hosted shopping cart. I have an existing website to promote my bricks and mortar retail operation and am expanding into web retailing. I will be using one of the major hosted shopping carts. What is the best way to join the two components from an SEO perspective? Have the cart as a sub domain of my main site, or move my existing domain name to be hosted by the cart provider and have both components operate under the same general domain? I have read arguments that putting your cart within a sub domain is not a good idea because any clout of the pre-existing domain will not be shared with the sub domain; that they will be treated as two separate sites. I have also read that using a sub domain is a good idea being that the content focus of the main domain (marketing and blogs) is different form the focus of the sub domain (product sales), and that the two components would benefit form earning their own rankings undiluted by the other. And, I have also read that search engines are getting good at being able to deduce that an eCommerce sub domain is legitimate extension of a content intensive main domain, and that they treat the two components as a combined whole. What is the truth? Which is the better way to go? Any guidance would be appreciated.
Technical SEO | | MEI1520 -
No indexing url including query string with Robots txt
Dear all, how can I block url/pages with query strings like page.html?dir=asc&order=name with robots txt? Thanks!
Technical SEO | | HMK-NL0 -
What are your best tips for SEO on a shopping cart?
So, I am working on a shopping cart platform (X-Cart) and so far don't like it. Also, the web designer is not someone I've worked with before and he is understandably conservative about access--which limits what I can and cannot do from the back end. One of the things I like to do is include text for the search engines. However, based on conversion, etc., I think the product images on a landing page (main brand info with specific products that show up) should show up first to move toward conversion first. I am thinking of adding the text below the product images on the brand pages so the viewer sees the products first while still keeping the content seo. My practice is to use between 300-350 words minimum on a page. Just wondering what best practices you have for a shopping cart. Care to share? Any tips or hints? Thoughts on what I might do that would be most effective? As always, thanks in advance for your sage advice!
Technical SEO | | TheARKlady0