New SEO manager needs help! Currently only about 15% of our live sitemap (~4 million url e-commerce site) is actually indexed in Google. What are best practices sitemaps for big sites with a lot of changing content?

[Hamish_TM]

In Google Search console

4,218,017 URLs submitted

402,035 URLs indexed

what is the best way to troubleshoot?

What is best guidance for sitemap indexation of large sites with a lot of changing content?

view?usp=sharing

[Nigel_Carr]

Hi Hamish

I'm not sure how many products you have listed on your website but I am only guessing that it is not 4m of even 400,000. I think the question you should be asking yourself is 'do I really need so many URLs?'

If you have 50,000 products in your site then frankly you only need maybe 51000 pages in total (including support pages, brands (maybe), categories and sub-categories.  I am only guessing but I would suggest that the other pages are being created by tags or other attributes and that these elements are creating acres of duplicate and very skinny content.

My usual question is - 'so you have 400,000 (never mind 4m) pages in Google? - did you write or generate 400,000  pages of useful, interesting, non-duplicate and shareable content? The answer of course is usually no.

Try switching off sets of tags and canonicalizing very similar content and you'll be amazed how it helps rankings!

Just a thought

Regards Nigel

Carousel Projects.

[DonnaDuncan]

This post from Search Engine Journal (https://www.searchenginejournal.com/definitive-list-reasons-google-isnt-indexing-site/118245/) is helpful for troubleshooting.

This Moz post (https://moz.com/blog/8-reasons-why-your-site-might-not-get-indexed) has some additional considerations. The 6th point the post author raises is one you should pay attention to given you're asking about a large e-commerce site. Point 6 says you might not have enough Pagerank, that "the number of pages Google crawls is roughly proportional to your pagerank".

As you probably know, Google has said they're not maintaining Pagerank anymore, but the essence of the issue raised is a solid one. Google does set a crawl budget for every website and large e-commerce sites often run into situations where they run out before the entire site is indexed. You should look at your site structure, robots tagging, and as Jason McMahon says, internal linking to make sure you are directing Google to the most important pages on your site first, and that all redundant content is canonicalized or noindexed.

I'd start with that.

[timcbambrick]

Hi  Hamish_TM,

It is hard to say without knowing the exact URL but here are some things to consider:

1. Indexing Lag - How long ago did you submit the sitemaps? We usually find there can be at least a few weeks lag between when the sitemaps are submitted and when all the URL's are indexed.
2. Internal Linking - What does your sites internal linking structure look like? Good internal linking like having breadcrumbs, in-text links, sidebar links and siloed URL structuring can help the indexation process.
3. **Sitemap Errors - **Are there currently any sitemap errors listed in Google Search Console? Either on the dashboard or in the sitemaps section? Any issues here could be adding to your problem.

Hopefully, this is of some help and let me know how you go.

Regards,

Jason.