Proper sitemap update frequency
-
I have 12 sitemaps submitted to Google. After about a week, Google is about 50% of the way through crawling each one.
In the past week I've created many more pages. Should I wait until Google is 100% complete with my original sitemaps or can I just go ahead and refresh them? When I refresh the original files will have different URLs.
-
You want your sitemap to include all your important URLs. Don't remove them from the sitemap just because you have been crawled.
-
Agreed, I don't see any issue with it, if you have more urls submit them if you can.
-
Nah, I don't think so. If they havent gotten to them yet, it shouldnt affect it. You could probably change the URL's, change the name of the sitemap, etc and have it not do anything.
If anything, you would want them to find the new URL's before its done with the first crawl, rather than index something that is no longer correct.
-
Thanks David. To clarify, the urls haven't changed I've just added more of them.
I am wondering if it will "throw google off" if I uploaded all new sitemaps that had different URLs in them before it's done with first crawl. I am getting good crawl frequency now and didn't want to disrupt it.
Does that make sense or change your answer at all?
Thanks again.
-
If you have URL's that changed, I would resubmit. If Google hasn't found them yet, what difference would it make to submit more that haven't been found yet? When they do crawl them, you will have them crawling the right and updated URL locations
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ruby on rails sitemap.xml structure
Is their a recommended way/best practice to implement sitemap.xml files on a site built with ruby on rails?
Intermediate & Advanced SEO | | brianvest0 -
How To Implement Pagination Properly? Important and Urgent!
I have seen many instructions but I am still uncertain. Here is the situation We will be implementing rel prev rel next on our paginted pages. The question is: Do we implement self referencing canonical URL on the main page and each paginated page? Do we implement noindex/follow meta robots tag on each paginated page? Do we include the canonical URL for each paginated page in the sitemap if we do not add the meta robots tag? We have a view all but will not be using it due to page load capabilities...what do we do with the viewl all URL? Do we add meta robots to it? For website search results pages containing pagination should we just put a noindex/follow meta robots tag on them? We have seperate mobile URL's that also contain pagination. Do we need to consider these pages as a seperate pagination project? We already canonical all the mobile URL's to the main page of the desktop URL. Thanks!
Intermediate & Advanced SEO | | seo320 -
Xml sitemap Issue... Xml sitemap generator facilitating only few pages for indexing
Help me I have a website earlier 10,000 WebPages were facilitated in xml sitemap for indexation, but from last few days xml sitemap generator facilitating only 3300 WebPages for indexing. Please help me to resolve the issue. I have checked Google webmaster indexed pages, its showing 8,141. I have tried 2-3 paid tools, but all are facilitating 3300 pages for indexing. I am not getting what is the exact problem, whether the server not allowing or the problem with xml sitemap generator. Please please help me…
Intermediate & Advanced SEO | | udistm0 -
Sort term product pages and fast indexing - XML sitemaps be updated daily, weekly, etc?
Hi everyone, I am currently working on a website that the XML sitemap is set to update weekly. Our client has requested that this be changed to daily. The real issue is that the website creates short term product pages (10-20 days) and then the product page URL's go 404. So the real problem is quick indexing not daily vs weekly sitemap. I suspect that daily vs weekly sitemaps may help solve the indexing time but does not completely solve the problem. So my question for you is how can I improve indexing time on this project? The real problem is how to get the product pages indexed and ranking before the 404 page shows u?. . Here are some of my initial thoughts and background on the project. Product pages are only available for 10 to 20 days (Auction site).Once the auction on the product ends the URL goes 404. If the pages only exist for 10 to 20 days (404 shows up when the auction is over), this sucks for SEO for several reasons (BTW I was called onto the project as the SEO specialist after the project and site were completed). Reason 1 - It is highly unlikely that the product pages will rank (positions 1 -5) since the site has a very low Domain Authority) and by the time Google indexes the link the auction is over therefore the user sees a 404. Possible solution 1 - all products have authorship from a "trustworthy" author therefore the indexing time improves. Possible solution 2 - Incorporate G+ posts for each product to improve indexing time. There is still a ranking issue here since the site has a low DA. The product might appear but at the bottom of page 2 or 1..etc. Any other ideas? From what I understand, even though sitemaps are fed to Google on a weekly or daily basis this does not mean that Google indexes them right away (please confirm). Best case scenario - Google indexes the links every day (totally unrealistic in my opinion), URL shows up on page 1 or 2 of Google and slowly start to move up. By the time the product ranks in the first 5 positions the auction is over and therefore the user sees a 404. I do think that a sitemap updated daily is better for this project than weekly but I would like to hear the communities opinion. Thanks
Intermediate & Advanced SEO | | Carla_Dawson0 -
Submitting XML Sitemap for large website: how big?
Hi there, I’m currently researching how I can generate an XML sitemap for a large website we run. We think that Google is having problems indexing the URLs based on some of the messages we have been receiving in Webmaster tools, which also shows a large drop in the total number of indexed pages. Content on this site can be accessed in two ways. On the home page, the content appears as a list of posts. Users can search for previous posts and can search all the way back to the first posts that were submitted. Posts are also categorised using tags, and these tags can also currently be crawled by search engines. Users can then click on tags to see articles covering similar subjects. A post could have multiple tags (e.g. SEO, inbound marketing, Technical SEO) and so can be reached in multiple ways by users, creating a large number of URLs to index. Finally, my questions are: How big should a sitemap be? What proportion of the URLs of a website should it cover? What are the best tools for creating the sitemaps of large websites? How often should a sitemap be updated? Thanks 🙂
Intermediate & Advanced SEO | | RG_SEO0 -
We have a static sitemap and a blog...Uh oh!
Hi We have a static sitemap and a blog...Uh oh! We are are just realizing that the 2 are not very compatible!! We need a dynamic sitemap. Is there such a thing as a "set and forget" sitemap that updates itself without us maintaining a database of sitemap data? Thanks Andrew
Intermediate & Advanced SEO | | Studio330 -
Updating Youtube Videos
My company has a popular Youtube channel. Some of the videos are old, and lower quality than the new HD videos we host on our site. We'd like to get the new, better videos on Youtube, but we don't want to erase the viewcounts and comments of the old videos. We also don't want a duplicate content penalty, if that exists on Youtube. What should we do? We could: 1. Overwrite our old videos with the new ones, and lose the view counts 2. Add the new videos, and leave the old videos untouched 3. Add the new videos, and add some kind of redirect from the old videos 4. Do nothing. We want to embed the new videos on our site to increase our Youtube view counts, so options 1-3 are preferable. Has anyone faced a similar situation?
Intermediate & Advanced SEO | | lighttable0 -
What is the practical influence of priority in a sitemap?
I have a directory site with 1000s of entries. Will there be benefit to be gained from playing with various entries priorities in the sitemap? I was thinking I might give more priority to entries that have upgraded their directory entry. Thanks.
Intermediate & Advanced SEO | | flow_seo0