Submitting sitemaps every 7 days
-
Question, if you had a site with more than 10 million pages (that you wanted indexed) and you considered each page to be equal in value how would you submit sitemaps to Google?
Would you submit them all at once: 200 sitemaps 50K each in a sitemap index?
Or
Would you submit them slowly? For example, would it be a good idea to submit 300,000 at a time (in 6 sitemaps 50k each). Leave those those 6 sitemaps available for Google to crawl for 7 days then delete them and add 6 more with 300,000 new links? Then repeat this process until Google has crawled all the links? If you implemented this process you would never at one time have more than 300,000 links available for Google to crawl in sitemaps.
I read somewhere that eBay does something like this, it could be bogus info though.
Thanks
David
-
Thanks Maurizio.
What I am really most concerned about is submitting hundreds of sitemaps to Google and giving them concern that we might be spamming them.
This is why I am considering the second approach where we would submit 6 sitemaps at a time which would total no more than 300,000 links rather than giving them 200 plus sitemaps with 10 million links.
I should have been clearer in my reason for this question. The main goal here is to not have Google freakout because we just gave them 10,000,000 links at one time.
-
hI
it's better divide the sitemap in many files, max 50k and create
how you can read in this page
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35738
"To fix this issue, break your Sitemap into several smaller Sitemaps, and list these in a Sitemap index file. (More information about Sitemap index files.) Upload your Sitemaps and Sitemap index files to your site, then submit these files individually."
Ciao
Maurizio
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Yoast XML Sitemap Taxonomies
Hey all, Quick question about taxonomies and sitemap settings. On our e-commerce site, we noindex product tags and post tags. Under the "Taxonomies" settings in Yoast I'm seeing taxonomies such as Product Color (pa_color). Would it be wise to remove such taxonomies from our sitemap if we already include product colors, attributes, etc. in our page titles and product descriptions? Thanks in advance, Andrew
Intermediate & Advanced SEO | | mostcg0 -
Lost homepage rankings for one day
We had a strange thing happen to our website. We have a website that ranks for top keywords for some years. Last week we lost alle of our rankings for one day. The strange thing was, that the rankings were only lost on the homepage, the homepage didnt rank anymore. Even when i googled for 'homepage.com' google showed 'homepage.com/page' and not the homepage. After some reading i checked for the following: no-index sources in the source code - but no results checked for http respons code - 200 ok status checked for downtime in pingdom - no downtime With no rankings on the homepage i was losing 90% of the traffic, i went to GWT and did a fetch as google request, the website looked okay. After that i request a new indexation request. After this the website was back in google with all the old rankings. But what has happend, is still a question for me. Can this be an hack or a wordpress problem or is this realtime penguin with a hit. I also read things about Google flux, but it didnt happen before and it was just the homepage, with no index at all. What happend?
Intermediate & Advanced SEO | | remkoallertz0 -
Sitemap Query
I've decided to write my own sitemap because frankly, the automated ones pull all kinds of out of I don't know where. So to get around that, manual it is. But I have some products appear in various categories, should I still list every product in each category in the sitemap, regardless of some being duplicates, or should I choose the most relevant category and list them there? I do have a canonical URL extension which should resolve any duplicate content I have.
Intermediate & Advanced SEO | | moon-boots0 -
Any excellent recommendations for a sitemap.xml plugin?
Hi, I'm trying to find a sitemap generator/plugin that I can point my client to. My client is using Magento, and is one of the largest sports store i Norway (around 20 000 products). I've heard there's one that can set the <priority>according to page views, sold units, and other relevant parameters, and that also takes care of the other elements in the sitemap.xml.</priority> Any good recommendations out there? 🙂
Intermediate & Advanced SEO | | Inevo0 -
Sitemap with homepage URL repeated several times - it is a problem?
Hello Mozzers, I am looking at a website with the homepage repeated several times (4 times) on the sitemap (sitemap is autogenerated via a plugin) - is this an SEO problem do you think - might it damage SEO performance, or can I ignore this issue? I am thinking I can ignore, yet it's an odd "issue" so your advice would be welcome! Thanks, Luke
Intermediate & Advanced SEO | | McTaggart0 -
Broken sitemaps vs no sitemaps at all?
The site I am working on is enormous. We have 71 sitemap files, all linked to from a sitemap index file. The sitemaps are not up to par with "best practices" yet, and realistically it may be another month or so until we get them cleaned up. I'm wondering if, for the time being, we should just remove the sitemaps from Webmaster Tools altogether. They are currently "broken", and I know that sitemaps are not mandatory. Perhaps they're doing more harm than good at this point? According to Webmaster Tools, there are 8,398,082 "warnings" associated with the sitemap, many of which seem to be related to URLs being linked to that are blocked by robots.txt. I was thinking that I could remove them and then keep a close eye on the crawl errors/index status to see if anything changes. Is there any reason why I shouldn't remove these from Webmaster Tools until we get the sitemaps up to par with best practices?
Intermediate & Advanced SEO | | edmundsseo0 -
Need Help Finding Directories to Submit To
I am looking for a lot of free "do follow" technology directories to submit to. Does anyone know of a good directory or a list of some sort of technology directories or something similar? Actually, I guess any directory that has a technology category would be helpful.
Intermediate & Advanced SEO | | MyNet0 -
Should the sitemap include just menu pages or all pages site wide?
I have a Drupal site that utilizes Solr, with 10 menu pages and about 4,000 pages of content. Redoing a few things and we'll need to revamp the sitemap. Typically I'd jam all pages into a single sitemap and that's it, but post-Panda, should I do anything different?
Intermediate & Advanced SEO | | EricPacifico0