Having Problems to Index all URLs on Sitemap
-
Hi all again ! Thanks in advance ! My client's site is having problems to index all its pages. I even bought the full extension of XML Sitemaps and the number of urls increased, but we still have problems to index all of them.
What are the reasons? The robots.txt is open for all robots, we only prohibit users and spiders to enter our Intranet. I've read that duplicate content and 404's can be the reason. Anything else?
-
Like already answered above it's quite hard to get to the 100% indexation rate for your webpages. What's your current indexation rate? If it's below 90% you might still have some issues somewhere.
-
Hi there
According to Google:
"...we don't guarantee that we'll crawl all of the pages of a particular site. Google doesn't crawl all the pages on the web, and we don't index all the pages we crawl. It's perfectly normal for not all the pages on a site to be indexed."
Google also provides tips and resources to help your site being indexed properly and, possibly, more fully. You can check that resource out here. Kissmetrics has a few other tips.
To Andy's point, Google indexes what it wants - don't be discouraged if your entire site isn't indexed in WMT. If you know how many pages are on your site (which you definitely should), I would try the "site:" function and get a better idea of what actually is indexed in Google.
Hope this helps! Good luck!
-
I have yet to see Google index every page of a site. They tend not to index pages that they don't think meet the criteria, so unless it was something like 90% weren't being indexed, I wouldn't worry about it. There are so many reasons that Google won't index a page.
You will also find that over time, if Google attributes more trust to the site, that more pages will be indexed.
Of course, you can do things to improve your changes, such as making sure Google can crawl all pages, check to see there are no bottlenecks anywhere and the big one - make sure your content is amazing. As long as the site is the best it can be, over time the number of indexed pages will increase.
Remember - the sitemap is not a guarantee that pages will be indexed. It just helps Google crawl your site.
-Andy
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problems with WooCommerce Product Attribute Filter URL's
I am running a WordPress/WooCommerce site for a client, and Moz is picking up some issues with URL's generated from WooCommerce product attribute filters. For example: ..co.uk/womens-prescription-glasses/?filter_gender=mens&filter_style=full-rim&filter_shape=oval How do I get Google to ignore these filters?
Technical SEO | | SushiUK
I am running Yoast Premium, but not sure if this can solve the issue? Product categories are canonicalised to the root category URL. Any suggestions very gratefully appreciated. Thanks Bob0 -
Do I need a link to my sitemap?
I have a very large sitemap. I submit it to both Google and Bing, but do I need a link to it? If someone went there it would probably lock their browser. Is there any danger of not having a link if I submit it to Google and Bing?
Technical SEO | | EcommerceSite0 -
How to determine which pages are not indexed
Is there a way to determine which pages of a website are not being indexed by the search engines? I know Google Webmasters has a sitemap area where it tells you how many urls have been submitted and how many are indexed out of those submitted. However, it doesn't necessarily show which urls aren't being indexed.
Technical SEO | | priceseo1 -
De-indexed from Google
Hi Search Experts! We are just launching a new site for a client with a completely new URL. The client can not provide any access details for their existing site. Any ideas how can we get the existing site de-indexed from Google? Thanks guys!
Technical SEO | | rikmon0 -
Walking into a site I didn't build, easy way to fix this # indexing problem?
I recently joined a team with a site without a) Great content b) Not much of any search traffic I looked and all their url's are built in this way: Normal looking link -> not actually a new page but # like: /#content-title And it has no h1 tag. Page doesn't refresh. My initial thought is to gut the site and build it in wordpress, but first have to ask, is there a way to make a site with /#/ content loading friendly to search engines?
Technical SEO | | andrewhyde0 -
Image Sitemap Indexing Issue
Hello Folks, I've been running into some strange issues with our XML Sitemaps. The XML Sitemaps won't open on a browser and it throws the following error instead of opening the XML Sitemap. Sample XML Sitemap - www.veer.com/sitemap/images/Sitemap0.xml.gzError - "XML Parsing Error: no element foundLocation: http://www.veer.com/sitemap/images/Sitemap0.xmlLine Number 1, Column 1:"2) Image files are not getting indexed. For instance, the sitemap - www.veer.com/sitemap/images/Sitemap0.xml.gz has 6,000 URLs and 6,000 Images. However, only 3,481 URLs and 25 images are getting indexed. The sitemap formatting seems good, but I can't figure out why Google's de-indexing the images and only 50-60% of the URLs are getting indexed. Thank you for your help!
Technical SEO | | CorbisVeer0 -
Blank Canonical URL
So my devs have the canonical URL loaded up on pages automatically, and in most cases this gets done correctly. However we ran across a bug that left some of these blank like so: Does anyone know what effect that would have? I am trying to provide a priority for this so I can say "FIX IT NOW" or "Fix it after the other 'FIX IT NOW' type of items". Let me know if you have any ideas. I just want to be sure I am not telling google that all of these pages are like the home page. Thanks!
Technical SEO | | SL_SEM0
