Using one robots.txt for two websites

[ciehmoz]

I have two websites that are hosted in the same CMS. Rather than having two separate robots.txt files (one for each domain), my web agency has created one which lists the sitemaps for both websites, like this:

User-agent: *
Disallow:

Sitemap: https://www.siteA.org/sitemap
Sitemap: https://www.siteB.com/sitemap

Is this ok? I thought you needed one robots.txt per website which provides the URL for the sitemap. Will having both sitemap URLs listed in one robots.txt confuse the search engines?

[ciehmoz]

Hi @gpainter,

Thanks for your help. I can't see anything specific in that link that says you can't have two sitemaps in one robots.txt. Where it mentions the sitemap it does say "You can specify multiple sitemap fields", although I'm not sure whether this means having multiple sitemap URLs under one mention of 'sitemap'?

[tir17]

@ciehmoz Hey I've replied to the other thread too.

The best case here will be to utilize different robots.txt files for both the websites.

You could've used the same robots.txt file only if the other site was on the same subdomain.

Don't forget to include the corresponding sitemaps to the new robots.txt file, hope this works out, cheers.

[GPainter]

Hey @ciehmoz

Just replied to your other thread, you will need one robot.txt per site. Referring to two sitemaps in one robots.txt will confuse Google.

Info here - https://developers.google.com/search/docs/advanced/robots/robots_txt

Good Luck