How do we handle sitemaps in robots.txt when multiple domains point to same physical location?
-
we have www.mysite.net, www.mysite.se, www.mysite.fi and so on. all of these domains point to the same physical location on our webserver, and we replace texts given back to client depending on which domain he/she requested.
My problem is this: How do i configure sitemaps in robots.txt when robots.txt is used by multiple domains? If I for instance put the rows
Sitemap: http://www.mysite.net/sitemapNet.xml
Sitemap: http://www.mysite.net/sitemapSe.xmlin robots.txt, would that result in some cross submission error?
-
Thanks for your help René!
-
yup
-
Yes, I mean GTW of course :).
A folder for each site would definitely make some things easier, but it would also mean more work every time we need to republish the site or make configurations.
Did I understand that googlelink correctly in that if we have verified ownership in GWT for all involved domains cross-site submission in robots.txt was okay? I guess google will think its okay anyway.
-
actually google has the answer, right here: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=75712
I always try to do what google recommends even though something might work just as well.. just to be on the safe side
-
you can't submit a sitemap in GA so I'm guessing you mean GWT
Whether or not you put it in the robots.txt shouldn't be a problem. since in each sitemap, the urls would look something like this:
Sitemap 1:<url><loc>http:/yoursite.coim/somepage.html</loc></url>
Sitemap 2:<url><loc>http:/yoursite.dk/somepage.html</loc></url>
I see no need to filter what sitemap is shown to the crawler. If your .htaccess is set-up to redirect traffic from the TLD (top level domain eg .dk .com ex.) to the correct pages. Then the sitemaps shouldn't be a problem.
The best solution would be: to have a web in web. (a folder for each site on the server) and then have the htaccess redirect to the right folder. in this folder you have a robots.txt and a sitemap for that specific site. that way all your problems will be gone in a jiffy. It will be just like managing different 3 sites. even though it isn't.
I am no ninja with .htaccess files but I understand the technology behind it and know what you can do in them. for a how to do it guide, ask google thats what I allways do when I need to goof around in the htaccess. I hope it made sense.
-
Thanks for your response René!
Thing is we already submit the sitemaps in google analytics, but this SEO company we hired wants us to put the sitemaps in robots.txt as well.
The .htaccess idea sounds good, as long as google or someone else dont think we are doing some cross-site submission error (as described here http://www.sitemaps.org/protocol.php#submit_robots)
-
I see no need to use robots.txt for that. use Google and Bings webmaster tools. Here you have each domain registered and can submit sitemaps to them for each domain.
If you want to make sure that your sitemaps are not crawled by a bot for a wrong language. I would set it up in the .htaccess to test for the entrance domain and make sure to redirect to the right file. Any bot will enter a site just like a browser so it needs to obey the server. so if the server tells it to go somewhere it will.
the robots.txt can't by it self, do what you want. The server can however. But in my opinion using bing and google webmaster tools should do the trick.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is my domain holding me back in the SERPS?
Even after a good year or so, my site intensivedrivingschoolmiltonkeynes.co.uk does not rank in the top 10 (google.co.uk) for "Intensive Driving School Milton Keynes", and is nowhere for "Driving School Milton Keynes". Do you think the domain name is being penalised, or do you think there are other factors that contribute to the poor performance.
On-Page Optimization | | Buffalo-Mobile1 -
One question about the domain choice.
Hi guys, I'm an Italian Web marketing and I'm studying SEO since some moths. First, sorry for my bad english.... 🙂 Second, the question: My Italian blog is going really GREAT in terms of popularity and incomes and know I want to bring it in english language with the help of a professional translator. The choice that I have to take is about the future domain of the english blog. My Italian blog has as url mynameandmysurname.net I don't really like to make the english blog appears like myitalianblogurl.net/english because I think it's a little bit confusing and not professional. I have also thought the idea to make a sub domain like blog.myitalianurl.net. What about the WPML plugin? Do you think it's a great choice? I don't like it because it's confusing and it affect my italian blog. Or maybe is it better to create a new branded blog with different domain? It can be an idea but I think that If I chose this way I will lose all my seo efforts of the last years 😄
On-Page Optimization | | Italianseolover0 -
Advice about resource points
Hi there, We're an online booking portal for the hair and beauty industry, and also provide salon management software. We're thinking of creating a separate subdomain on the main site, probably along the lines of salons.example.com as a resource point and advice centre for salons. In it, I was thinking of featuring links and extracts from various blog posts I've written on salon marketing, downloadable white papers and featured videos we've created - a sample of each that you can click through to get all the bits on one page i.e. you click a see all vids bit to be taken to a page with all the vids on it. This is very much conceptual at the moment (though we do have a fair number of resources on the blog and the like), but I'd love it all to be drawn into a helpful section for salons. I'd love to hear what you think about it and whether there are some other ways we might look into doing this. No right answers, hoping for a bit of a debate/advice around resource centres. Thanks a million!
On-Page Optimization | | LeahHutcheon0 -
Why isn't our site being shown on the first page of Google for a query using the exact domain, when its pages are indeed indexed by Google
When I type our domain.com as a query into Google, I only see one of our pages on the homepage, and it's in 4th position. It seems though, that all pages of the site are indexed by google when I type in the query "site:domain.com". There was an issue at the site launch, where the robots.txt file was left active for around two weeks. Would this have been responsible for the fact that another domain ranks #1 when we type in our own domain? It has been around a couple of months now since the site was launched. Thanks in advance.
On-Page Optimization | | featherseo0 -
On Page Reports - Multiple URLs Appearing for a Keyword
Hello, I have a question regarding the on page reports automatically generated by seomoz When I look at my on page reports I notice that each keyword appears a number of times, each with a different url and then a grade for the on page report and sometimes a rank. I'm not sure I understand this, firstly I thought the on page reports were only generated for keywords in the top 50, does that mean the global top 50, or my top 50? Also why are they appearing for so many urls, I find this confusing and am not sure which pages to focus on improving, it's not always my intended pages that are ranking the best. I believe that I read somewhere that I can choose which pages to have the on page reports rank for, perhaps this is the solution? Any thoughts would be appreciated. Thanks, Iain
On-Page Optimization | | jannkuzel0 -
Our sitemap is not indexed well
Hey there, Hope you guys can help. We get the following error: Nested indexing. Another Sitemap index refers to the index of sitemaps. The thing is that we cant find the error they are talking about. Thanks!!!!
On-Page Optimization | | Comunicare0 -
Looking to Buy a New Domain for 10 Years
I want to Know, for how much year i should buy a new domain for optimal SEO Results. Should I Buy it for 3 Years, 5 Years or 10 Years
On-Page Optimization | | Ganesh340 -
Link Juice not passing for root domain to an internal page
Hi Im working in a SEO proyect for the web fagorarrasate.com, it has good root domain ranking, but it seems like the root domain isn't passing any of its link juice to other internal pages of the site although there are links which link to these sub pages For example if I seach in Google for the page http://www.google.com/search?aq=f&gcx=c&sourceid=chrome&ie=UTF-8&q=Stamping+System+and+Presses The firs page appears without any page ranking, so it seems that the root domain is not passing any of its ranking Why could this be happening? Thanks Jordi
On-Page Optimization | | joralles0