Why might Google be crawling via old sitemap, when the new one has been submitted and verified?
-
We have recently relaunched Scoutzie.com and re-submitted our new sitemap to Google. When I look on Webmaster tools, our new sitemap has been submitted just fine, but at the same time, Google is finding a lot of 404s when crawling the site. My understanding, it is still using crawling the old links, which do not exists. How can I tell Google to refresh it's index and to stop looking at all the old links?
-
Yes it should. However, as Alan mentioned below, if you still have links pointing to the 404 pages, Google will always attempt to crawl them, and will keep you informed that you have errors.
If you do have external links to those 404 pages, you can 301 redirect them to an appropriate page using .htaccess. This way you'll keep the link value and also get rid of the Webmaster Tools error.
If you don't have any links to them, then yes, Google will eventually stop trying to crawl them.
-
It's very likely that we do. Given that I cannot track down a 1000+ links that now 404, will they eventually fall out by themselves, or do I have to tell Google that everything that's 404'ed should be dropped from crawl index? Thanks!
-
What if I simply pushed the new sitemap over the old one? In other words, scoutzie.com/sitemap is the same link, except now it contains the new map. That should be okay, right?
-
you may still have links pointing to those 404 pages on your site or externally. If not then eventually they will fall out of the index
-
Hey scoutzie,
This is actually covered pretty well in Joe Robison's blog post on fixing Webmaster Tools crawl errors: http://moz.com/blog/how-to-fix-crawl-errors-in-google-webmaster-tools
I'll quote the related info:
"One frustrating thing that Google does is it will continually crawl old sitemaps that you have since deleted to check that the sitemap and URLs are in fact dead. If you have an old sitemap that you have removed from Webmaster Tools, and you don’t want being crawled, make sure you let that sitemap 404 and that you are not redirecting the sitemap to your current sitemap."
Hope this helps, good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to add a Google+ business page to the social analytics?
Since my Google+ account is attached to my business page, MOZ is tracking the actual user page. I want to track the business page. It won't let me add another user. How do I fix this? Thanks.
Moz Pro | | kcampbell990 -
Where is the crawl test tool located in this new site?
Hi there, Where is the crawl test tool located in this new moz site? Formerly it was, http://pro.seomoz.org/tools/crawl-test Hope you can help. Thanks:)
Moz Pro | | steveovens1 -
Does Google Reward Bad Links in Some Industries?
This week I've been diving into competitive research again for several clients in different industries, and unless the data in Open Site Explorer is wrong, it really does appear that Google continues to reward sites with unnatural links in certain industries, especially industries that are less tech-savvy (agriculture, finance, manual labor). I've combed through all the links carefully, to ensure there aren't some secret high-quality links lurking, and for some of these sites that are showing up in the top ten for search they do not even one real, high-quality link. Also, their onsite optimization is often way over the top and the user experience is nothing less than nauseating. Could it be a MozScape issue, or does Google continue to reward terrible sites and link structures in certain industries? I'd love to hear your thoughts.
Moz Pro | | newwhy0 -
Why does Crawl Diagnostics report this as duplicate content?
Hi guys, we've been addressing a duplicate content problem on our site over the past few weeks. Lately, we've implemented rel canonical tags in various parts of our ecommerce store, over time, and observing the effects by both tracking changes in SEOMoz and Websmater tools. Although our duplicate content errors are definitely decreasing, I can't help but wonder why some URLs are still being flagged with duplicate content by our SEOmoz crawler. Here's an example, taken directly from our Crawl Diagnostics Report: URL with 4 Duplicate Content errors:
Moz Pro | | yacpro13
/safety-lights.html Duplicate content URLs:
/safety-lights.html ?cat=78&price=-100
/safety-lights.html?cat=78&dir=desc&order=position /safety-lights.html?cat=78 /safety-lights.html?manufacturer=514 What I don't understand, is all of the URLS with URL parameters have a rel canonical tag pointing to the 'real' URL
/safety-lights.html So why is SEOMoz crawler still flagging this as duplicate content?0 -
Tools that crawl 2 million page sites
Our site is about 2million pages deep, 50% of which is stale content. Yes, I know - OMG #unhygienic. Even if we get approval to get rid of half of it. SEOMoz Pro Elite only crawls 20k deep - what can i do to crawl and diagnose the whole site. Are there any tools anyone can suggest. SEOMoz??
Moz Pro | | ilhaam0 -
Has MozTrust predicted fall in Google rankings???
The definition of MozTrust is:
Moz Pro | | driansmith
"MozTrust is SEOmoz's global link trust score. It is similar to MozRank but rather than measuring link popularity, it measures link trust. Receiving links from sources which have inherent trust, such as the homepages of major university websites or certain governmental web pages, is a strong trust endorsement." That being the case it is quite disturbing that a number of websites have been hit very badly by the latest Google algorithm changes that did indeed have very respectable MozTrust rankings. Can I ask whether anybody has carried out any sort of analysis of MozTrust versus negative impact in Google rankings? The prediction would of course be that websites suffering from a lower MozTrust would have been hit quite badly by these recent changes.0 -
Crawl diagnostic Notices for rel Canonical increased
Hello, We just signed up for SEO Moz, and are reviewing the results of our second web crawl. Our Errors and Warnings summary have been reduced, but our Notices for Rel Canonical have skyrocketed from 300 to over 5,500. We are using a WP with the Headway theme and our pages already have the rel=canonical along wiht rel=author. Any ideas why this number would go up so much in one week? Thank you, Michael
Moz Pro | | MKaloud0 -
How to crawl the whole domain?
Hi, I have a website an e-commerce website with more than 4.600 products. I expect that Seomoz scan check all url's. I don't know why this doesn't happens. The Campaign name is Artigos para festa and should scan the whole domain festaexpress.com. But it crels only 100 pages I even tried to create a new campaign named Festa Express - Root Domain to check if it scans but had the same problem it crawled only 199 pages. Hope to have a solution. Thanks,
Moz Pro | | EduardoCoen
Eduardo0