Sitemap Indexed vs. Submitted
-
My sitemap has been submitted to Google for well over 6 months and is updated frequently, a total of 979 URLs have been submitted by only 145 indexed. What can I do to get Google to index them all?
-
SF finding 'useless' links is actually part of its purpose, if you believe they're useless you should be asking why they're there. Your XML sitemap should have nothing but clean URLs; 200 response codes and not canonicalized to another URL. The problem isn't that you have category URLs, it's that those (like the one in my previous example) have a canonical tag that points elsewhere. Anytime this is the case, the URL is considered un-indexable. You can see the proof of this by doing a Google search for "https://www.interstellarstore.com/meteorite-jewelry/meteorite-necklaces", I just checked and this URL isn't in the index.
You mentioned the age in your original comment that your XML sitemap had been submitted for well over 6 months, that's where I got the age from, maybe I misunderstood?
You have no reason to not trust SF, it's one of the most valuable tools in an SEO's toolbox. I've used it for 5+ years to create hundreds of sitemaps and countless other SEO tasks with no problem in providing reliable, accurate data points.
-
Hi Logan,
I tried using Screaming Frog but it kept finding useless links, so I wrote the sitemap myself and I update it manually, I updated it only this morning. What makes you think it is over 6 months since an update?
I was told on Moz in an earlier post that having all of the category links, not just the canonical ones, wasn't a problem, is this not the case?
Every link in the sitemap should work fine, I wrote it by copy and pasting the links directly from my site. I have no trust in Screaming Frog.
-
Hi,
I poked around a bit on your sitemap and noticed a couple things:
- You've got URLs on there that have canonicals to another page. For example:This page https://www.interstellarstore.com/meteorite-jewelry/meteorite-necklaces has a canonical tag that points here https://www.interstellarstore.com/meteorite-necklaces.
- A bunch of the URLs in your sitemap redirect elsewhere or have no response - I got 13% through crawling your XML sitemap with Screaming Frog and there were zero 200 response code URLs, not good.
Both of these things combined are causing a discrepancy in the amount of submitted URLs vs. indexed URLs. If you use Screaming Frog to create your XML sitemap it's quite easy to have only clean URLs in there. You can easily remove all URLs that are not 200 status and by default Screaming Frog will exclude any URL that canonicalizes to another URL.
Also, as a side note, you should be updating your XML sitemap more frequently, a 6 month old sitemap for an ecommerce site is far too old with new products being added and products dropping off.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I no-index categories of my blog?
I have blog with lots of articles & it also has lots of categories. These categories are currently indexed in the google and moz showing missing title and description for these categories. Should I place no-index tag in all the categories or leave it as it is?
Intermediate & Advanced SEO | | jhakasseo0 -
Pending Sitemaps
Hi, all Wondering if someone could give me a pointer or two, please. I cannot seem to get Google or Bing to crawl my sitemap. If I submit the sitemap in WMT and test it I get a report saying 44,322urls found. However, if I then submit that same sitemap it either says Pending (in old WMT) or Couldn't fetch in the new version. This couldn't fetch is very puzzling as it had no issue fetching the map to test it. My other domains on the same server are fine, the problem is limited to this one site. I have tried several pages on the site using the Fetch as Google tool and they load without issue, however, try as I may, it will not fetch my sitemap. The sitemapindex.xml file won't even submit. I can confirm my sitemaps, although large, work fine, please see the following as an example (minus the spaces, of course, didn't want to submit and make it look like I was just trying to get a link) https:// digitalcatwalk .co.uk/sitemap.xml https:// digitalcatwalk .co.uk/sitemapindex.xml I would welcome any feedback anyone could offer on this, please. It's driving me mad trying to work out what is up. Many thanks, Jeff
Intermediate & Advanced SEO | | wonkydogadmin0 -
Fix Google Index error
I changed my blog URL structure Can Someone please let me how to solve this?
Intermediate & Advanced SEO | | Michael.Leonard0 -
How to Submit My new Website in All Search Engines
Hello Everyone, Can Any body help to suggest Good software, or Any other to easily Submit my website , to All Search Engines ? ? Any expert Can help please, Thanx in Advance
Intermediate & Advanced SEO | | falguniinnovative0 -
To index or not to index search pages - (Panda related)
Hi Mozzers I have a WordPress site with Relevanssi the search engine plugin, free version. Questions: Should I let Google index my site's SERPS? I am scared the page quality is to thin, and then Panda bear will get angry. This plugin (or my previous search engine plugin) created many of these "no-results" uris: /?s=no-results%3Ano-results%3Ano-results%3Ano-results%3Ano-results%3Ano-results%3Ano-results%3Akids+wall&cat=no-results&pg=6 I have added a robots.txt rule to disallow these pages and did a GWT URL removal request. But links to these pages are still being displayed in Google's SERPS under "repeat the search with the omitted results included" results. So will this affect me negatively or are these results harmless? What exactly is an omitted result? As I understand it is that Google found a link to a page they but can't display it because I block GoogleBot. Thanx in advance guys.
Intermediate & Advanced SEO | | ClassifiedsKing0 -
Member request pages, indexed or no indexed?
We run a service website and basically users of the site post their request to get certain items fixed/serviced. Through Google Analytics we have found that we got lots of traffic to these request pages from people searching for those particular items. E.g. A member's request page: "Cost to fix large Victorian oven" has got many visits from searchers searching for "large Victorian oven". The traffic to these pages is about 40% of our Google organic traffic but didn't covert to more users/requests well and has roughly 67% bounce rate. So my question is: should we keep these pages indexed and if yes what can we do to improve the conversion rate/reduce bounce rate? Many thanks guys. David
Intermediate & Advanced SEO | | sssrpm0 -
How do I increase rankings when the indexed page is the homepage?
Hi Forum, This is a two-part question. The first is: "what may be the cause of some rank declines?" and the second is "how do I bring them back up when the indexed page is the homepage?" Over the last week I noticed some declines in several of my top keywords, many of which point to the site's homepage. The site itself is an eCommerce site, which had less visits last week than normal (holidays it seems, since the data jibes with key dates). Can a decline in traffic cause ranking declines? Any other ideas of where to look? Secondly, for those keywords that link to the homepage, how do we bring these back up since a homepage can't be optimized for every single keyword? We sell yoga products and can't have a homepage that is optimized for keywords like "yoga mat," "yoga blocks," "yoga pilates clothing," and several others, as these are our category pages' keywords. Any thoughts? Thanks!
Intermediate & Advanced SEO | | pano0 -
How long till pages drop out of the index
In your experience how long does it normally take for 301-redirected pages to drop out of Google's index?
Intermediate & Advanced SEO | | bjalc20110