Getting More Pages Indexed

[Mattchstick]

We have a large E-commerce site (magento based) and have submitted sitemap files for several million pages within Webmaster tools. The number of indexed pages seems to fluctuate, but currently there is less than 300,000 pages indexed out of 4 million submitted. How can we get the number of indexed pages to be higher? Changing the settings on the crawl rate and resubmitting site maps doesn't seem to have an effect on the number of pages indexed.

Am I correct in assuming that most individual product pages just don't carry enough link juice to be considered important enough yet by Google to be indexed? Let me know if there are any suggestions or tips for getting more pages indexed.

syGtx.png

[Boodreaux]

I think that is what did it! lol

[EGOL]

Yes, you will need internal links to establish your site navigation.  Then, external links if you don't have enough PR flow from within your site.

Some powerful sites can support these millions of pages with internal links.  If you have a site like that congratulations!

[Mattchstick]

Thanks this is helpful. I will work on funneling spiders. I assume I'll need a healthy dose of both internal and external links pointing deep into the site in order to get the spider to start chewing in there? Thanks.

[EGOL]

Thanks!   You enjoyed the chewing spiders?

[Boodreaux]

Wow...that was a powerful answer EGOL.  Thanks for adding the long answer.  The way you worded it created a visualization for me that was very helpful to my understanding as a novice.  Thanks.

[EGOL]

The short answer...

Link deep into the site at multiple points with heavy PR.

The long answer...

If you have a really big site you need a lot of linkjuice to get the entire site indexed and keep it in the index.  You also need a good site structure so that spiders can crawl through the site and find every page.

If you have several million pages, my guess is that you will need hundreds of links of at least PR5 or PR6 linking into the site.  I would direct each of those links to a deep category page.  That will funnel the spiders deep into your site and force them to chew their way out while indexing your pages.

All of those links must be held permanently in place.  Because if you pull the links the flow of spiders will stop and google will slowly forget about pages that are not visited by spiders on a regular basis.

If you have weak links or not enough links your site will not be thoroughly crawled and google will forget about your pages as fast as they are discovered.

Big sites require a PR resource.