Webmaster Tools Indexed pages vs. Sitemap?
-
Looking at Google Webmaster Tools and I'm noticing a few things, most sites I look at the number of indexed pages in the sitemaps report is usually less than 100% (i.e. something like 122 indexed out of 134 submitted or something) and the number of indexed pages in the indexed status report is usually higher. So for example, one site says over 1000 pages indexed in the indexed status report but the sitemap says something like 122 indexed.
My question: Is the sitemap report always a subset of the URLs submitted in the sitemap? Will the number of pages indexed there always be lower than or equal to the URLs referenced in the sitemap?
Also, if there is a big disparity between the sitemap submitted URLs and the indexed URLs (like 10x) is that concerning to anyone else?
-
Unfortunately not, the closest you'll get is selecting a long period of time in Analytics and then exporting all the pages that received organic search traffic. If you could then cross check them with your list of URLs on your site it could provide you with a small list. But I would still check them in Google to make sure they aren't indexed. As I said it's not the best way.
-
Is there a reliable way to determine which pages have not been indexed?
-
Great answer by Tom already, but I want to add that probably images and other types of content whom are mostly not by default included in sitemaps could also be among the indexed 'pages'.
-
There's no golden rule that your sitemap > indexed pages or vice versa.
If you have more URLs in your sitemap than you have indexed pages, you want to look at the pages not indexed to see why that is the case. It could be that those pages have duplicate and/or thin content, and so Google is ignoring them. A canonical tag might be instructing Google to ignore them. Or the pages might be off the site navigation and are more than 4 links/jumps away from the homepage or another page on the site, make them hard to find.
Conversely, if you had lots more pages indexed than in your sitemap, it could be a navigation or URL duplication problem. Check to see if any of the pages are duplicate versions caused by things like dynamic URLs generated through search on the site or the site navigation, for example. If those pages are the only physical pages that you have created and you know every single one has been submitted in a sitemap - and so any other indexed URLs would be unaccounted for, that may well be cause for concern, so check nothing is being indexed multiple times.
Just a couple of scenarios, but I hope it helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On Google Analytics, Pages that were 301 redirected are still being crawled. What's the issue here?
URL that we redirected are being crawled on Google Analytics. Since they dont exist, they have high bounce rates. What can the issue be?
Reporting & Analytics | | prestigeluxuryrentals.com0 -
Bing Webmaster Tools data discrepency on traffic
Hi, For a given time period in Bing Webmaster Tools it says that combined clicks from Bing and Yahoo organic search are about 4,700 total, for the same time period in Google Analytics, combining sessions for Bing and Yahoo Organic comes out to about 8,900 total. Has anyone else experienced this discrepancy? Is this common? How can I get 8900 visits from 4700 clicks?
Reporting & Analytics | | IrvCo_Interactive0 -
How is it possible that this site has a higher page authority than my site?
Judging by open site explorer, I'm crushing my competitor in every imaginable way. And yet, somehow they have a higher page authority than me and, consequently, are ranking higher than me. How is this possible? My site is on the left: 40atcpP.png
Reporting & Analytics | | ScottMcPherson0 -
Moz is showing different "errors" than Webmaster tools
I have set up my Moz campaign and the crawl errors are showing multiple duplicate content and page titles however when I check my webmaster tools data, these errors are not showing up. Is this normal and who should I listen to?
Reporting & Analytics | | LabelMedia0 -
800,000 pages blocked by robots...
We made some mods to our robots.txt file. Added in many php and html pages that should not have been indexed. Well, not sure what happened or if there was some type of dynamic conflict with our CMS and one of these pages, but in a few weeks we checked webmaster tools and to our great surprise and dismay, the number of blocked pages we had by robots.txt was up to about 800,000 pages out of the 900,000 or so we have indexed. 1. So, first question is, has anyone experienced this before? I removed the files from robots.txt and the number of blocked files has still been climbing. Changed the robots.txt file on the 27th. It is the 29th and the new robots.txt file has been downloaded, but the blocked pages count has been rising in spite of it. 2. I understand that even if a page is blocked by robots.txt, it still shows up in the index, but does anyone know how the blocked page affects the ranking? i.e. while it might still show up even though it has been blocked will google show it at a lower rank because it was blocked by robots.txt? Our current robots.txt just says: User-agent: *
Reporting & Analytics | | TheCraig
Disallow: Sitemap: oursitemap Any thoughts? Thanks! Craig0 -
Google.co.uk (The Web or Pages From UK) Query?
Hi, Google.co.uk is ambiguous at best, it is geo targeted for the UK, however, by default all results incorporate "The Web" meaning outside the UK. If a user wishes to filter to "Pages From UK" then they have to click that specifically. Now my clients regularly ask me whether the traffic they are getting is from Google.co.uk (The Web) or Google.co.uk (Pages from UK) In analytics it combines these two as single source = Google.co.uk without any further breakdown, is there a way to figure this out. If I can split the figures then I can run necessary additional comparisons etc. Regards Ausaf
Reporting & Analytics | | conversiontactics0 -
How long does it take to get results from the Term Extractor Tool?
Hi! I entered a web page into the Term Extractor Tool, and it's been running for about 3 minutes now. How long does it usually take? Thanks 🙂 Kelley
Reporting & Analytics | | kinsana0 -
Google vs Bing/Yahoo SERP results?
I see large differences in Google and Bing/Yahoo SERP results for many keywords. Google shows many of our primary keywords in their top ten, while Bing/Yahoo rank the same one 40-50 or above. Do you have any insight on their differences that might account for this variation? We are legitimate, long time, white hatters at a small manufacturer that is one of the leaders in our industry. The only thing I can think of that might describe this issue is PPC advertising. Their algorithms wouldn't be that inconsistent would they? (We do much more paid advertising on Google) But generally this should have no affect, right? Thanks, George...
Reporting & Analytics | | rhawk0