Automated XML Sitemap for a BIG site

[mattdinbrooklyn]

Hi,

I would like to do an automated sitemap for my site but it has more than a million pages. It would need to be a sitemap index with a separation on different parts of the site (i.e. news, video) and I'll want a news sitemap and video sitemap as well (of course). Does anyone have any recommended way of making this and how much would you recommend it getting updated? For news and , I would like it to be pretty immediate if possible but the static pages don't need to be updated as much.

Thanks!

[CleverPhD]

Another good reference:

http://googlewebmastercentral.blogspot.com/2014/10/best-practices-for-xml-sitemaps-rssatom.html

that points to how to ping:

http://www.sitemaps.org/protocol.html#submit_ping

specific search engine examples:

http://codingtips.kanishkkunal.in/ping-sitemap/

[mattdinbrooklyn]

Excellent. Thank you! How would you ping google when a sitemap is updated?

[CleverPhD]

Yes, split them out.  You will need an index sitemap. That is a sitemap that links to other sitemaps

https://support.google.com/webmasters/answer/75712?vid=1-635768989722115177-4024498483&rd=1

In any given sitemap you can have up to 50,000 URLs listed in it and it can be no larger than 50MB uncompressed.

https://support.google.com/webmasters/answer/35738?hl=en&vid=1-635768989722115177-4024498483

Therefore, you could have an index sitemap with links up to 50,000 other sitemaps.  Each of those sitemaps could contain links to 50,000 URLs on your site each.

If my math is right, that would be a max of 2,500,000,000 URLs if you have 50,000 sitemaps of 50,000 URLs each.

(Interesting side note Google allows up to 500 index sitemaps, so if you take 2,500,000,000 pages x 500 - 1,250,000,000,000 URLs that you can submit to Google via sitemaps)

How you divide up your content into sitemaps would relate to how your organize the pages on your site, so you are on the right track in breaking out the sitemaps by types of content.  Depending on how big any one section of the site is, you may need to have more of those sitemaps in that type i.e. articlesitemap1.xml articlesitemap2.xml etc.  You get the idea.

It is recommended that you ping Google every time a page in a sitemap is updated so Google will come back and recrawl the sitemap.  I don't run any sites with 1M URLs but I do run several that run in the 10s of thousands.  We break them up by type and ping whenever we update a page in that group.  You need to consider your crawl budget with Google in that it may not crawl all 1M pages in your sitemap as often and so you may consider for a group of pages setting them up so that if you have articlesitemap1.xml, articlesitemap2.xml, articlesitemap3.xml you are always adding your newest URLs to the most recent sitemap created (i.e. articlesitemap3.xml)  That way you are generally pinging Google about the update of a single sitemap out of the group vs all three.

My other thought is that in addition to pinging Google only on the sitemaps that that you have updated, you show a 304 server response to all sitemaps that have not been updated.  304 means "not modified" since last visit.  One of your challenges will be your crawl budget with Google and so why make them recrawl a sitemap they have already crawled?  You may want to consider a 304 on any URL on your site that has not changed since last time Google visited.

All of that said, as I mentioned above, I have not worked at the scale of 1M+ pages and would defer to others on the best way to approach.  The general thought process would be the same though in trying to figure out the best way to use your sitemaps to manage your crawl budget from Google.  Small side note, if you have 1M+ pages and any of those are from the use of things like sorting parameters, duplicate content, printer friendly pages, you may want to just noindex them regardless and leave them out of the sitemap and not allow Google to crawl them to start with.