Submitting XML Sitemap for large website: how big?
-
Hi there,
I’m currently researching how I can generate an XML sitemap for a large website we run. We think that Google is having problems indexing the URLs based on some of the messages we have been receiving in Webmaster tools, which also shows a large drop in the total number of indexed pages.
Content on this site can be accessed in two ways. On the home page, the content appears as a list of posts. Users can search for previous posts and can search all the way back to the first posts that were submitted.
Posts are also categorised using tags, and these tags can also currently be crawled by search engines. Users can then click on tags to see articles covering similar subjects. A post could have multiple tags (e.g. SEO, inbound marketing, Technical SEO) and so can be reached in multiple ways by users, creating a large number of URLs to index.
Finally, my questions are:
- How big should a sitemap be? What proportion of the URLs of a website should it cover?
- What are the best tools for creating the sitemaps of large websites?
- How often should a sitemap be updated?
Thanks
-
Thanks Matt, that's really useful
-
Yeah, it's better to have one than not - but I have always aimed to make it as complete as I can. Why? I'm not sure - mostly because I figure Google is GREAT at crawling my main structure - it's those far-reaching pages that I'm hoping they find in the sitemap.
-
Thanks for both your replies - I will check out the tools and recommendations you suggested.
I'm sure I remember somewhere reading a recommendation that it was only necessary to submit the basic site structure in a sitemap. It sounds like this is not the case and that a site map should , if possible, be comprehensive.
Would it be better to have a basic sitemap giving the main navigational URLs than having nothing at all?
-
I've created sitemaps with the paid version of Screaming Frog that were almost 80,000 pages. That's what I'd use. No point asking what % unless you can't get it all. If you're crawling Microsoft, break it up. Otherwise, organize it if you can (category sitemap, month by month, something.) or just make one big finger to Google type sitemap. lol
-
Hi!
First off, since your content can be accessed in multiple ways, I'd make sure that you're applying means to indicate duplicate pages as such to search engines. Easy access to great content is fantastic, but you can devaluate your own pages a lot when you're not careful. If you're not using it yet, I recommend implementing the rel="canonical" tag in your website.
To answer your questions:
- It should cover all URLs that want indexed. Ideally, that would be every URL
- I'm not sure what 'the best' tools would be, but I used http://www.xml-sitemaps.com a lot a few years back. Their sitemaps are free up to 500 URLs. There are payment plans for bigger ones.
- I wouldn't update an XML sitemap for every new page you make once a month. Instead, let the search engine find their own way in that case. Should your entire site structure change, an XML sitemap can be a great way to help search engine understand your new site setup better.
I hope this helps!
- It should cover all URLs that want indexed. Ideally, that would be every URL
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
International XML Sitemaps - Standalone, or Integrate into Existing XML Sitemap?
Hi there, We understand that hreflang tagging can be incorporated into an existing XML sitemap. That said, is there any inherent issue with having two sitemaps - your regular XML sitemap plus an international XML sitemap which lists off many of the same URLs as your original XML sitemap? For example, one of our clients has an XML sitemap file they don't want to have to edit, but we want to implement international hreflang xml sitemaps for them. Can we add an "English" XML sitemap with the proper hreflang tagging even though this new sitemap contains many duplicates as the existing XML sitemap file? Thank you!
Intermediate & Advanced SEO | | FPD_NYC0 -
This url is not allowed for a Sitemap at this location error using pro-sitemaps.com
Hey, guys, We are using the pro-sitemaps.com tool to automate our sitemaps on our properties, but some of them give this error "This url is not allowed for a Sitemap at this location" for all the urls. Strange thing is that not all of them are with the error and most have all the urls indexed already. Do you have any experience with the tool and what is your opinion? Thanks
Intermediate & Advanced SEO | | lgrozeva0 -
What is the difference between Multilingual and multiregional websites?
Hi all, So, I have studied about multilingual and multiregional websites. As soon as possible, we will expand the website languages to english and spanish. The urls will be like this: http://example.com/pt-br
Intermediate & Advanced SEO | | mobic
http://example.com/en-us
http://example.com/es-ar Thereby, the tags will be like this: Great! But my doubt is: To /es-ar/ The indexing will be only to spanish languages in Argentina? What about the other countries that speak the same language, like Spain, Mexico, etc.I don't know if it will be possible develop a Spanish languages especially for each region. Should I do an multiregional website or only multilingual? How Google sees this case? Thanks for any advice!!1 -
How ot optimise a website for competitive keywords?
Hi guys, I hope to find some good answers to my questions, because here are some of the best SEO's in the world. I'm doing SEO as a hobby for a few years and had some very good results before the latest Google updates. Now I'm not able to rank any website for competitive keywords. The last project I started is this website (man and van hire company targeting London market).
Intermediate & Advanced SEO | | nasi_bg
The problem is that I can't rank even in Top 100 in Google UK for the main keywords like: "man and van london" , "man and van service london" ,"london man & van"...
The site has over 1k good backlinks (according to Ahrefs), unique content, titles and descriptions but still can't rank well. Am i missing something? Few years back that was more than enough to rank well in Google.
I will be very grateful to hear your suggestions and opinions.0 -
Priority Attribute in XML Sitemaps - Still Valid?
Is the priority value (scale of 0-1) used for each URL in an XML sitemap still a valid way of communicating to search engines which content you (the webmaster) believe is more important relative to other content on your site? I recall hearing that this was no longer used, but can't find a source. If it is no longer used, what are the easiest ways to communicate our preferences to search engines? Specifically, I'm looking to preference the most version version of a product's documentation (version 9) over the previous version (version 8). Thanks!
Intermediate & Advanced SEO | | Allie_Williams0 -
Problems with a website-help
Soooooo, I did a crawl report on this site : www.greatwesternflooring.com and this was what was on the report. This is a dnn site. I'm guessing the site has a redirect loop given the http status code. Can anyone help me with a fix. (the developers have said there is no redirect on the site......clearly there is....) | http://www.greatwesternflooring.com/ | 2015-01-07T21:32:25Z | 609 : Redirect to already-visited URL received for page request. | Error attempting to request page; see title for details. | 302 | http://www.greatwesternflooring.com | <colgroup><col width="319"> <col width="144"> <col width="378"> <col span="39" width="64"></colgroup>
Intermediate & Advanced SEO | | Britewave
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |0 -
Domain and Sitemap Question
Hi - I am hoping you can help me with this issue we are currently trying to solve. We are hosting our mobile site's content on a different domain than what the URL of the site is, though owned by same company. In Google Webmasters tool we have the mobile sitemap under "sitemaps.xyz.com", however the URL of the site is "m.xyz.com". We have submitted 60MM pages in the mobile sitemap, but only 1MM pages have been indexed. Do you think this set up causes confusion with the bots? Does this affect the crawlability of the site? Any thoughts would be greatly appreciated. Thank you!
Intermediate & Advanced SEO | | ladylana
Eva0 -
Should subdomains be avoided for brand new websites?
When creating a brand new website, will setting it up as a subdomain provide ranking benefits? I understand that if it's an existing domain, it's better to use a subfolder because a subdomain is treated as a different domain. But is there any reason not to start a website with the keyword in the subdomain? For example: keyword.domain.com The SERP's are dominated by websites which contain some variation of the head term, but the disadvantage of doing a similar this is your website looks very similar. Thanks!
Intermediate & Advanced SEO | | JonDavies540