Using one robots.txt for two websites
-
I have two websites that are hosted in the same CMS. Rather than having two separate robots.txt files (one for each domain), my web agency has created one which lists the sitemaps for both websites, like this:
User-agent: * Disallow: Sitemap: https://www.siteA.org/sitemap Sitemap: https://www.siteB.com/sitemapIs this ok? I thought you needed one robots.txt per website which provides the URL for the sitemap. Will having both sitemap URLs listed in one robots.txt confuse the search engines?
-
Hi @gpainter,
Thanks for your help. I can't see anything specific in that link that says you can't have two sitemaps in one robots.txt. Where it mentions the sitemap it does say "You can specify multiple sitemap fields", although I'm not sure whether this means having multiple sitemap URLs under one mention of 'sitemap'?
-
@ciehmoz Hey I've replied to the other thread too.
The best case here will be to utilize different robots.txt files for both the websites.
You could've used the same robots.txt file only if the other site was on the same subdomain.
Don't forget to include the corresponding sitemaps to the new robots.txt file, hope this works out, cheers.
-
Hey @ciehmoz
Just replied to your other thread, you will need one robot.txt per site. Referring to two sitemaps in one robots.txt will confuse Google.
Info here - https://developers.google.com/search/docs/advanced/robots/robots_txt
Good Luck
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Adding your sitemap to robots.txt
Hi everyone, Best practice question: When adding your sitemap to your robots.txt file, do you add the whole sitemap at once or do you add different subcategories (products, posts, categories,..) separately? I'm very curious to hear your thoughts!
Technical SEO | | WeAreDigital_BE0 -
Multiple robots.txt files on server
Hi! I have previously hired a developer to put up my site and noticed afterwards that he did not know much about SEO. This lead me to starting to learn myself and applying some changes step by step. One of the things I am currently doing is inserting sitemap reference in robots.txt file (which was not there before). But just now when I wanted to upload the file via FTP to my server I found multiple ones - in different sizes - and I dont know what to do with them? Can I remove them? I have downloaded and opened them and they seem to be 2 textfiles and 2 dupplicates. Names: robots.txt (original dupplicate)
Technical SEO | | mjukhud
robots.txt-Original (original)
robots.txt-NEW (other content)
robots.txt-Working (other content dupplicate) Would really appreciate help and expertise suggestions. Thanks!0 -
Ecommerce website - Transition from one subdomain to another. Please help!
Hi MOZers, I have an ecommerce website with its mobile version of the site sitting in a subdomain. It is going to be transferred to another subdomain on the same website. How do I ensure that I save/carry over most of the traffic, authority and equity to the new sudomain? At the moment, we are not looking to get rid of the subdomain but maybe later, yes. Malika
Technical SEO | | Malika10 -
Images on Website for SEO
Good Morning, We have a magento website with hundreds of different products that have slight size variations. The image for each of these products looks the same (the only difference between the products is some of the dimensions) .... Would you recommend using the same image for each of these products and just use a generic file name that describes the overall product or would you give each product its own image with it's specific product name as the file name? Should I use 1 image for 500 different sku's or should i rename the file the name of each individual sku and load an individual image? The end user will not know the difference since all of the images will appear identical, simply asking from an SEO perspective. Thanks
Technical SEO | | Prime850 -
"One Page With Two Links To Same Page; We Counted The First Link" Is this true?
I read this to day http://searchengineland.com/googles-matt-cutts-one-page-two-links-page-counted-first-link-192718 I thought to myself, yep, thats what I been reading in Moz for years ( pitty Matt could not confirm that still the case for 2014) But reading though the comments Michael Martinez of http://www.seo-theory.com/ pointed out that Mat says "...the last time I checked, was 2009, and back then -- uh, we might, for example, only have selected one of the links from a given page."
Technical SEO | | PaddyDisplays
Which would imply that is does not not mean it always the first link. Michael goes on to say "Back in 2008 when Rand WRONGLY claimed that Google was only counting the first link (I shared results of a test where it passed anchor text from TWO links on the same page)" then goes on to say " In practice the search engine sometimes skipped over links and took anchor text from a second or third link down the page." For me this is significant. I know people that have had "SEO experts" recommend that they should have a blog attached to there e-commence site and post blog posts (with no real interest for readers) with anchor text links to you landing pages. I thought that posting blog post just for anchor text link was a waste of time if you are already linking to the landing page with in a main navigation as google would see that link first. But if Michael is correct then these type of blog posts anchor text link blog posts would have value But who is' right Rand or Michael?0 -
Two domains, one unique design?
Hi Mozzers, There are lots of articles on using multiple domains for a company and we all know the story behind that. I have a different question though - I designed and built a really great WordPress theme for my company and it runs on xyz.com and has run on their for well over a year and has decent authority. I would now like to expand my business and offer a different service and I'd like to market the new service on abc.com My question is: If I use unique content on the second domain, but the same WordPress theme, will it have a negative SEO effect on either site? I should mention: I have my reasons for not just using the same domain for both services. I'd love to see what everyone thinks 🙂
Technical SEO | | ChristopherM0 -
Redirection in two phases
Hi, One of my client want a website revamp in two phases. The first phase would be to move the CMS from Sharepoint to Drupal keeping the same content and URLs but the page extension will change (it could become php or just the name of the page without any extension) The second phase will be a content revamp with URL change. The first phase will last 3 weeks and then we will push the second phase with the new content and the new URL. Some shortcuts to make it a more readable: old url = OURL old url with new extension = OURLEX new URL = NURL My problem is how can I manage the redirections. Should I: Phase 1: create one rule that will redirect all the OURL to the OURLEX Phase 2: keep the rule I created in phase 1 and add rules for OURLEX to NURL Or Phase 1: create one rule that will redirect all the OURL to the OURLEX Phase 2: keep the rule I created in phase 1 and add redirect OURL to NURL Or Phase 1: create one rule that will redirect all the OURL to the OURLEX Phase 2: remove the rule I created in phase 1 and add redirect OURL to NURL Or Phase 1: create a rules for each OURL that will redirect all the pages to OURLEX Phase 2: remove the rules I created in phase 1 and redirect the OURL to NURL Or Phase 1: create a rules for each OURL that will redirect all the pages to the OURLEX Phase 2: keep the rules I created in phase 1 and add rules to redirect the OURLEX to the NURL Or something else you think is better Difficult to explain, let me know if it's understandable. Thanks for you help! GaB
Technical SEO | | Pherogab0 -
Is blocking RSS Feeds with robots.txt necessary?
Is it necessary to block an rss feed with robots.txt? It seems they are automatically not indexed (http://googlewebmastercentral.blogspot.com/2007/12/taking-feeds-out-of-our-web-search.html) And, google says here that it's important not to block RSS feeds (http://googlewebmastercentral.blogspot.com/2009/10/using-rssatom-feeds-to-discover-new.html) I'm just checking!
Technical SEO | | nicole.healthline0
