Sitemaps and Indexed Pages
-
Hi guys,
I created an XML sitemap and submitted it for my client last month.
Now the developer of the site has also been messing around with a few things.
I've noticed on my Moz site crawl that indexed pages have dropped significantly.
Before I put my foot in it, I need to figure out if submitting the sitemap has caused this.. can a sitemap reduce the pages indexed?
Thanks
David.
-
Thanks Eli!
I guess I was wondering if the MOZ Bot only followed pages that were in the sitemap. It was generated by Screaming Frog I have trusted it to include all relevant pages!
I have put in a more detailed description in the response below. Overall I need to investigate further but i'm satisfied that the sitemap has not caused the drop!
-
Thanks Martijn!
I guess I was wondering if the MOZ Bot only followed pages that were in the sitemap. It was generated by Screaming Frog I have trusted it to include all relevant pages!
To elaborate.
There were about 80,000 pages and I used canonical, no index, and redirects to clean up a rather large mess of filter URL's and dup content.
That dropped the pages to about 14k. Then I submitted the sitemap last month and now the crawl only found 4k pages.
Further investigation is needed on my behalf but I wanted to double check that this sudden drop was not because of a sitemap! Thanks for clarifying that!
-
Hi David,
Messing up, Changing or Updating, Deleting a Sitemap is not necessarily something that will decrease the number of ranked or crawled pages. It usually is used a signal to find new pages and figure out if old ones are deleted. But the chances that your sitemap have had a significant impact in what kind of pages went down is something I would find unlikely. It could happen though that you'd see the opposite, an increase in pages indexed/submitted/crawled after you submit a sitemap.
Martijn.
-
Hey David!
Thanks for reaching out to us!
Unfortunately I am not an SEO consultant / Web Developer so I cannot offer specific advice, but I'm sure there are loads of members here who would love to help and have a lot more knowledge than I do! A few things I have picked up which may help are the following:
Try to determine when the drop started, did it drop when you submitted the XML sitemap or when the developer changed certain things? This could help point to the reason for the drop in indexing. There are a variety of reasons as to why Google may not choose to index pages, however some of the common ones are:
-
Check your robots.txt to ensure those pages are still crawlable
-
Check for duplicate content / was there any canonical changes?
-
One of the tools you could use to help keep track of ranking fluctuations is mozcast (http://mozcast.com/). Was there turbulence in the Google algorithm when the indexed pages dropped significantly?
If you want us to have a look at your specific campaign to investigate further could you please pop an email over to help@moz.com.
Thanks!
Eli
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sitemaps and Indexed Pages
Hi guys, I created an XML sitemap and submitted it for my client last month. Now the developer of the site has also been messing around with a few things. I've noticed on my Moz site crawl that indexed pages have dropped significantly. Before I put my foot in it, I need to figure out if submitting the sitemap has caused this.. can a sitemap reduce the pages indexed? Thanks David.
API | | Slumberjac0 -
Still not got any index update data.
Is anyone finding that they haven't got the results of the update yet? I have tried some competitors and they are not updated either.
API | | AHC_SEO0 -
Does on-page grader have an API ?
Hi, I would very much like to include the on-page grader output into my SEO tools. Is there an API for that? thanks James
API | | KMdayJob0 -
3 result limit to Top Pages API call
I am using the MOZ API to make calls for the top pages for a particular URL. However, when I pass in any limit value greater than 3 the API only returns 3 results. I have even tried to put in URLs like 'www.moz.com' and still only 3 results. Sample call to the API below: http://lsapi.seomoz.com/linkscape/top-pages/www.moz.com?AccessID=member-xxxxxxxxx&Expires=1419020831&Signature=xxxxxxxxx&Cols=2052&Offset=0&Limit=50
API | | solodev0 -
On Page Grader Problem-Sorry But This Page Inaccessible
Greetings: When I try to use the on page grader and enter my URL, an error message appears stating: "Sorry But This Page Inaccessible". The URL is http://www.nyc-officespace-leader.com/commercial-space/office-space and it works fine when I enter it on my browser. Any page from this domain generates this error. Is there a bug with this tool? How would I go about tracking ranking on various keywords? I see it is possible to tag keywords, and I have done so for about 250. But I don't know how to generate a ranking report for these keywords; ideally I would like to do so filtering them by the label I have applied. Any suggestions? Thanks,
API | | Kingalan1
Alan0 -
On page reports
Hi everyone I have just been going through the online page I see that I have quite a few words that have an F rating I was wondering if I have all the keywords with an A ranking would that improve our Moz rating? Also of the below elements can anyone tell which, if any are more important than the others? Title URL Meta Desc H1 H2-4 Body B / Strong IMG ALT
API | | Hardley1110 -
On-Page Reports showing old urls
While taking a look at our sites on-page reports I noticed some of our keywords with very old urls that haven't existed for close to a year. How do I make sure moz's keyword ranking is finding the correct page and make sure I'm not getting graded on that keywords/urls that don't exist any more or have been 301'd to new urls? Is there a way to clean these out? My on-page reports say I have 62 reports for only a total of 34 keywords in rankings. As you can see from the image most of the urls for "tax folder" have now been 301'd to not include /product or /category but moz is still showing them with the old url structure. BTW our site is minespress.com 2KdGcPL.png
API | | smines0 -
Top Pages metrics in OSE
Not sure if this is an API question or a feature request, but wondering if other folks had a way to do this: In OSE there is the dashboard for a specific URL that is entered into the search bar, giving you metrics at a glance on it. But I often find myself going to the Top Pages tab to get a sense of the domain as a whole. First off, wouldn't it be nice/is there a way to build my own "dashboard" based on info from that section? Specifically, I'd love to see at a glance the number of "top pages" that exist (many websites are well under the 10,000 page limit for this section, but there's no quick glance metric showing that) One thing that would be very handy for me would be a breakdown of HTTP Status info across the whole domain, as being able to see the raw total of different statuses (and the percentages of each based on the total number of pages) would be really helpful, giving me a sense if I should dig into any issues before exporting the list to CSV. I've found myself needing this type of info for multiple domains at once, so what would be REALLY cool would be a Google Doc where I could paste in different domains in one column, and this info being returned in other columns. I've searched through the Q&A and didn't find anything like this, and I didn't know how easy/hard any of this would be to do, but I was wondering if anyone else had a sense of how to solve this problem and how feasible it would be to tackle it. Thanks!
API | | John-E-Turner0
