Why extreme drop in number of pages indexed via GWMT sitemaps?

[jkinnisch]

Any tips on why our GWMT Sitemaps indexed pages dropped to 27% of total submitted entries (2290 pages submitted, 622 indexed)?  Already checked the obvious Test Sitemap, valid URLs etc. We had typically been at 95% of submitted getting indexed.

[jkinnisch]

Thanks, that coves it!

[DarinPirkey]

Yes, this is the norm.  You will generally have a variety of update frequencies in your xml sitemap.  If you look at your sitemap you will usually see a value from 0.1 to 1.0.  Those request the frequency in which the page is updated.  If Googlebot will generally adhere to your guidelines and only crawl those pages when you tell them they are updated.  If all of your pages are set to the same frequency, which they shouldn't be, Google will generally only crawl a certain amount of data on your site on a given crawl.  So, a slow increase in indexed pages is the norm.

[jkinnisch]

Yes, looking back at change logs was helpful. Canonical tags was it!  We found a bug, the canonical page tags were being truncated at 8 characters.  The number of pages indexed has started to increase rather than decrease, so it appears the issue is resolved.  But I would have thought the entire sitemap would get indexed once the issue was resolved, rather than small increases each day. Does that seem correct to have a slow increase back to normal, rather than getting back to nearly 100% indexed overnight?

[DarinPirkey]

Do you have the date of the change?  Try to see if you can see the when the change happened because we might be able to figure it out that way too.

WMT > sitemaps > webpages tab

Once you find the date you may be able to go through your notes and see if you've done anything around that date or if Google had any sort of update (PageRank just updated).

I have had sites that had pages unindexed and then a few crawls later it got reindexed.  I just looked at 20 sites in our WMT and all of our domains look good as far as percentage of submitted vs indexed.

Only other things I can think of is to check for duplicate content, canonical tags, noindex tags, pages with little or no value (thin content) and (I've done this before) keep your current sitemap structure but add an additional sitemap with all of your pages and posts to it.  Don't break it down, just add it all to one sitemap.  I've had that work before for a similar issue but that was back in 2010.  Multiple sitemaps for that site never seemed to work out.  Having it all on one did the trick.  The site was only about 4,000 pages at the time but I thought I would mention it.  I haven't been able to duplicate the error and no other site has had that problem but that did do the trick.

Definitely keep an eye on it over the next few crawls.  Please let us know what the results are and what you've tried so we can help troubleshoot.

[jkinnisch]

We use multiple site maps.
Thanks, I had not thought about page load speed. But it turned up okay. Had already considered your other suggestions. Will keep digging.  Appreciate your feedback.

[DarinPirkey]

Not sure why the drop but are you using just one sitemap or do you have multiple ones?

Check the sizes of your pages and the crawl rate that Google is crawling your site.  If they have an issue with the time it takes them to crawl your sitemap, it will start to reduce the number of indexed pages it serves up.   You can check your crawl stats by navigating to WMT, crawl > crawl stats.  Check to see if you've notice any delays in the numbers.

Also, make sure that your robots.txt isn't blocking anything.

Have you checked your site with a site: search?

These are pretty basic stuff but let us know what you've looked into so we can help you more.  Thanks.