Google Webmaster Tools Sitemap errors for phantom urls?
-
Two weeks ago we changed our urls so the correct addresses are all lowercase. Everything else 301 redirects to those. We have submitted and made sure that Google has downloaded our updated sitemap several times since.
Even so, Webmaster Tools is reporting 33000 + errors in our sitemap for urls that are no longer in our sitemap and haven't been for weeks. It claims to have found the errors within the last couple of days but the sitemap has been updated for a couple of weeks and has been downloaded by Google at least three times since.
Here is our sitemap: http://www.aquinasandmore.com/urllist.xml
Here are a couple of urls that Webmaster Tools says are in the sitemap:
http://www.aquinasandmore.com/catholic-gifts/Caroline-Gerhardinger-Large-Sterling-Silver-Medal/sku/78664
Redirect errorunavailable
Oct 7, 2011
http://www.aquinasandmore.com/catholic-gifts/Catherine-of-Bologna-Small-Gold-Filled-Medal/sku/78706
Redirect errorunavailable
Oct 7, 2011 -
How long does the actual data usually take to catch up with what WMT says is current?
I have not experienced any delay before. There should only be one sitemap record for your site at any time. That record could be composed of multiple files, but it is one collection of records.
When Google identifies crawl errors, those errors should be generated from the sitemap on file at the time of the error. There is a view sitemap option in Google WMT you can use to see the sitemap they have on file. This step would be next. If you can confirm the bad URL does not appear in the sitemap, I would then wait to see if the issue re-appears after today, October 11th.
I know this is frustrating but the system is very straight forward. I cannot explain why a URL not included in your sitemap would appear on your sitemap crawl errors tab. The only two possibilities I can come with is either you have made an error when sharing some information, or there is an unusual glitch on Google's end.
With all the above noted, working with sitemaps is not a good investment of your time. If your site navigation is properly designed, your sitemap offers no benefit whatsoever.
-
"then these links should not appear going forward." - They are showing up now even though Google says they have our latest sitemap and that the errors were found yesterday. How long does the actual data usually take to catch up with what WMT says is current?
The image urls are built from the actual title on the fly and don't 301 so those aren't a problem. The other one you mentioned does need to be cleaned up in the site map. Thanks for catching that.
These errors are showing up when I go to the crawl errors section and click the sitemap tab. Yes, the sitemap I shared is the same one in WMT.
-
I was unable to locate the URLs listed in your sitemap. If you Google WMT tools settings are correct and the sitemap which you have shared is the same one listed in your Google WMT account, then these links should not appear going forward.
You would need to examine your Google WMT account closely to determine the exact source of these errors.
Where exactly within your Google WMT are you seeing these errors? How are you identifying the source of these URLs are being from your sitemap?
Two weeks ago we changed our urls so the correct addresses are all lowercase.
There are many URLs in your site map which are not lower case. An example:
http://www.aquinasandmore.com/title/Brian-Kolodiejchuk/FuseAction/store.AuthorSearch/Author/2337/
Also you share a lot of image URLs which are not lower case either.
I would not necessarily advise cleaning up the entire site, but at least establish the best practice going forward.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Practice Approaches to Canonicals vs. Indexing in Google Sitemap vs. No Follow Tags
Hi There, I am working on the following website: https://wave.com.au/ I have become aware that there are different pages that are competing for the same keywords. For example, I just started to update a core, category page - Anaesthetics (https://wave.com.au/job-specialties/anaesthetics/) to focus mainly around the keywords ‘Anaesthetist Jobs’. But I have recognized that there are ongoing landing pages that contain pretty similar content: https://wave.com.au/anaesthetists/ https://wave.com.au/asa/ We want to direct organic traffic to our core pages e.g. (https://wave.com.au/job-specialties/anaesthetics/). This then leads me to have to deal with the duplicate pages with either a canonical link (content manageable) or maybe alternatively adding a no-follow tag or updating the robots.txt. Our resident developer also suggested that it might be good to use Google Index in the sitemap to tell Google that these are of less value? What is the best approach? Should I add a canonical link to the landing pages pointing it to the category page? Or alternatively, should I use the Google Index? Or even another approach? Any advice would be greatly appreciated. Thanks!
Intermediate & Advanced SEO | | Wavelength_International0 -
Google Indexing
Hi We have roughly 8500 pages in our website. Google had indexed almost 6000 of them, but now suddenly I see that the pages indexed has gone to 45. Any possible explanations why this might be happening and what can be done for it. Thanks, Priyam
Intermediate & Advanced SEO | | kh-priyam0 -
Editing A Sitemap
Would there be any positive effect from editing a site map down to a more curated list of pages that perform, or that we hope they begin to perform, in organic search? A site I work with has a sitemap with about 20,000 pages that is automatically created out of a Drupal plugin. Of those pages, only about 10% really produce out of search. There are old sections of the site that are thin, obsolete, discontinued and/or noindexed that are still on the sitemap. For instance, would it focus Google's crawl budget more efficiently or have some other effect? Your thoughts? Thanks! Best... Darcy
Intermediate & Advanced SEO | | 945010 -
Does Google Read URL's if they include a # tag? Re: SEO Value of Clean Url's
An ECWID rep stated in regards to an inquiry about how the ECWID url's are not customizable, that "an important thing is that it doesn't matter what these URLs look like, because search engines don't read anything after that # in URLs. " Example http://www.runningboards4less.com/general-motors#!/Classic-Pro-Series-Extruded-2/p/28043025/category=6593891 Basically all of this: #!/Classic-Pro-Series-Extruded-2/p/28043025/category=6593891 That is a snippet out of a conversation where ECWID said that dirty urls don't matter beyond a hashtag... Is that true? I haven't found any rule that Google or other search engines (Google is really the most important) don't index, read, or place value on the part of the url after a # tag.
Intermediate & Advanced SEO | | Atlanta-SMO0 -
Google webmaster tools showing "no data available" for links to site, why?
In my google webmaster account I'm seeing all the data in other categories except links to my site. When I click links to my site I get a "no data available" message. Does anyone know why this is happening? And if so, what to do to fix it? Thanks.
Intermediate & Advanced SEO | | Nicktaylor10 -
301 forwarding old urls to new urls - when should you update sitemap?
Hello Mozzers, If you are amending your urls - 301ing to new URLs - when in the process should you update your sitemap to reflect the new urls? I have heard some suggest you should submit a new sitemap alongside old sitemap to support indexing of new URLs, but I've no idea whether that advice is valid or not. Thanks in advance, Luke
Intermediate & Advanced SEO | | McTaggart0 -
Whole site blocked by robots in webmaster tools
My URL is: www.wheretobuybeauty.com.auThis new site has been re-crawled over last 2 weeks, and in webmaster tools index status the following is displayed:Indexed 50,000 pagesblocked by robots 69,000Search query 'site:wheretobuybeauty.com.au' returns 55,000 pagesHowever, all pages in the site do appear to be blocked and over the 2 weeks, the google search query site traffic declined from significant to zero (proving this is in fact the case ).This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster toolsrobots.txt file existed but did not have any entries to allow or disallow URLs - today I have removed robots.txt file completely URL re-direction within Linux .htaccess file - there are many rows within this complex set of re-directions. Developer has double checked this file and found that it is valid.I have read everything that google and other sources have on this topic and this does not help. Also checked webmaster crawl errors, crawl stats, malware and there is no problem there related to this issue.Is this a duplicate content issue - this is a price comparison site where approx half the products have duplicate product descriptions - duplicated because they are obtained from the suppliers through an XML data file. The suppliers have the descriptions from the files in their own sites.Help!!
Intermediate & Advanced SEO | | rrogers0 -
Subdirectory URLs
If I have category pages for my site; is it better to use http://example.com/category/category or just http://example.com/category? Also, I'm creating a new section of the site; a resource center. Should the URLs of the pages in the resource center be http://example.com/learn/page or just http://example.com/page What are the reasons for the better choice?
Intermediate & Advanced SEO | | Visually0