Why might Google be crawling via old sitemap, when the new one has been submitted and verified?
-
We have recently relaunched Scoutzie.com and re-submitted our new sitemap to Google. When I look on Webmaster tools, our new sitemap has been submitted just fine, but at the same time, Google is finding a lot of 404s when crawling the site. My understanding, it is still using crawling the old links, which do not exists. How can I tell Google to refresh it's index and to stop looking at all the old links?
-
Yes it should. However, as Alan mentioned below, if you still have links pointing to the 404 pages, Google will always attempt to crawl them, and will keep you informed that you have errors.
If you do have external links to those 404 pages, you can 301 redirect them to an appropriate page using .htaccess. This way you'll keep the link value and also get rid of the Webmaster Tools error.
If you don't have any links to them, then yes, Google will eventually stop trying to crawl them.
-
It's very likely that we do. Given that I cannot track down a 1000+ links that now 404, will they eventually fall out by themselves, or do I have to tell Google that everything that's 404'ed should be dropped from crawl index? Thanks!
-
What if I simply pushed the new sitemap over the old one? In other words, scoutzie.com/sitemap is the same link, except now it contains the new map. That should be okay, right?
-
you may still have links pointing to those 404 pages on your site or externally. If not then eventually they will fall out of the index
-
Hey scoutzie,
This is actually covered pretty well in Joe Robison's blog post on fixing Webmaster Tools crawl errors: http://moz.com/blog/how-to-fix-crawl-errors-in-google-webmaster-tools
I'll quote the related info:
"One frustrating thing that Google does is it will continually crawl old sitemaps that you have since deleted to check that the sitemap and URLs are in fact dead. If you have an old sitemap that you have removed from Webmaster Tools, and you don’t want being crawled, make sure you let that sitemap 404 and that you are not redirecting the sitemap to your current sitemap."
Hope this helps, good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is a new website the only way to get better rankings?
Our website is https://www.franklin-bell.com.au. For a while now we've been trying to get a few of our pages on there to get a better rank on google. A local SEO agency told us that some of our pages are not like "the modern standard" for web pages, but our page score for the keywords we want is 90% and higher. Do we really need to completely rebuild the website (it was built in 2014) in order to get better rankings? Or is it sufficient to have the 97% page score and then just get some back links and social media referrals etc. Thanks in advance.
Moz Pro | | franklin-bell0 -
Why is MOZ and Google search Volume so different?
A search term in MOZ shows the monthly search volume to be 49K. In Google, the same term shows the search volume at only 1300 monthly searches. Which do I trust? Thanks, Don
Moz Pro | | rcman0 -
OSE vs google webmaster link data
Hiya experts, I am trying to understand the OSE. One thing I noticed is that OSE shows 12 linking root domains to our site. However, google webmasters show more than 90(that have kinks to pages on our site). And these are not new links, some of these date back to Apr 2011. Is there something very obvious I am missing here? Thanks for your help. Regards, Raman.
Moz Pro | | ramangarg0 -
Can Google see all the pages that an seomoz crawl picks up?
Hi there My client's site is showing around 90 pages indexed in Google. The seomoz crawl is returning 1934 pages. Many of the pages in the crawl are duplicates, but there are also pages which are behind the user login. Is it theoretically correct to say that if a seomoz crawl finds all the pages, then Google has the potential to as well, even if they choose not to index? Or would Google not see the pages behind the login? And how come seomoz can see the pages? Many thanks in anticipation! Wendy
Moz Pro | | Chammy0 -
It would be Great if their were further integrations with Google Webmaster Tools
It would be great if their were plans to integrate Google Webmaster Tools into the mix. Specifically the Errors section. I am currently working on a new Campaign where I am seeing a little bit of overlap, but Google is finding all sorts of different missing pages from 3 redesigns ago but also quite a few current ones. Currently in SEOmoz: 0 Errors, while Google is reporting 12 - 403 Errors for some content the client unpublished. While the addition of Google Analytics was a nice, it would be great to dig further into Webmaster tools and Analytics with features that discover errors and provide actionable next steps. Is anyone else seeing these discrepancies between SEOmoz and Google Webmaster Tools?
Moz Pro | | drewschug0 -
Old Incoming Links Redirected to new pages are not Being Factored on the Open Site Explorer
Hi, My website has been online since 1994. We have old links pointing to pages that no longer exist so what we have done is to create redirects to the specific page where the content is being displayed now. However, when we use the Open Site Explorer, the linking root domains do not show those sites that contain old links pointing to our domain. One good example of this one is: dmoz.org. Am I improperly handling the redirects? Or, what do I have to do so that old links that are being redirected to the new pages where the information is are accounted when calculating domain authority and trust? Thanks, Alex
Moz Pro | | costarica.com0 -
Links listed in MozPro Crawl Diagnostics
Ok, seeing as I'm getting to the end of my first week as a Pro Member, I'm getting more and more feedback regarding the pages on my site. I'm slightly concerned though that, having logged in this morning, I'm being shown 407 warnings for pages with 'Too Many On Page Links.' According to the blurb at the top of the page, 'Too Many' is generally defined as being over 100 links on a page ... but when I look at the pages which are being thrown up in the report, none of them contain anywhere near 100 links. I seriously doubt there is a glitch with the tool which has led me to think that maybe there's an issue with the way my site is coded. Is anyone aware of a coding problem that may lead Google and SEOMoz to suspect that I have a load of links across my site? P.S. As an aside, when this tool mentions 'Too Many Links' is it referring purely to OBL or does it count links to elsewhere on my domain too? Cheers,
Moz Pro | | theshortstack0 -
Question about when new crawls start
Hi everyone, I'm currently using the trial of seomoz and I absolutely love what I'm seeing. However, I have 2 different websites (one has over 10,000 pages and one has about 40 pages). I've noticed that the smaller website is crawled every few days. However, the larger site hasn't been crawled in a few days. Although both campaigns state that the sites won't be crawled until next Monday, is there any way to get the crawl to start sooner on the large site? The reason that I've asked is that I've implemented some changes that will likely decrease the amount of pages that are crawled simply based upon the recommendations on this site. So, I'm excited to see the potential changes. Thanks, Brian
Moz Pro | | beeneeb0