Reasons for a sharp decline in pages crawled
-
Hello!
I have a site I've been tracking using Moz since July. The site is mainly stagnant with some on page content updates. Starting the first week of December, Moz crawler diagnostics showed that the number of pages crawled decreased from 300 to 100 in a week.
So did the number of errors through. So crawler issues went from 275 to 50 and total pages crawled went from 190 to 125 in a week and this number has stayed the same for the last 5 weeks.
Are the drops a red flag? Or is it ok since errors decreased also? Has anyone else experienced this and found an issue?
FYI: sitemap exists and is submitted via webmaster tools. GWT shows no crawler errors nor blocked URLs.
-
Google is indexing just over 80 URLs, although about 40% of them are developer test URLs (they lead to live pages of the site though). Nothing in robots.txt. No errors.
The Google bot is still crawling, but it's crawling half the pages. What would make it decrease in page crawls? I'm working if there is a broken link or something on the home page that's pointing away from the site... although it's unlikely, I'll check....
-
If you fixed a problem, such as duplicate content, that would mean that we're showing fewer errors and crawling fewer pages, since that problem is fixed. Might that be the case?
-
How many URLs are indexed in Google if you use site:yourdomain.com Has that figure dropped too?
Have you got anything in your robots.txt that could be blocking?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will more landing pages with related keywords improve ranking of main keyword?
Hi all, We are in the process of creating new landing pages and noticed something what our competitors been following with landing pages. They have created multiple landing pages targeting the related keywords of a main topic / main keyword. Like "seo defined", "seo history", "seo benefits", "seo strategies", "types of seo", etc being "SEO" as main keyword or main topic. Will this works? Thanks
Algorithm Updates | | vtmoz0 -
Sitemaps for landing pages
Good morning MOZ Community, We've been doing some re-vamping recently on our primary sitemap, and it's currently being reindexed by the search engines. We have also been developing landing pages, both for SEO and SEM. Specifically for SEO, the pages are focused on specific, long-tail search terms for a number of our niche areas of focus. Should I, or do I need to be considering a separate sitemap for these? Everything I have read about sitemaps simply indicates that if a site has over 50 thousand pages or so, then you need to split a sitemap. Do I need to worry about a sitemap for landing pages? Or simply add them to our primary sitemap? Thanks in advance for your insights and advice.
Algorithm Updates | | bwaller0 -
Duplicate pages in language versions, noindex in sitemap and canonical URLs in sitemap?
Hi SEO experts! We are currently in the midst of reducing our amount of duplicate titles in order to optimize our SEO efforts. A lot of the "duplicate titles" come from having several language versions of our site. Therefore, I am wondering: 1. If we start using "" to make Google (and others) aware of alternative language versions of a given site/URL, how big a problem will "duplicate titles" then be across our domains/site versions? 2. Is it a problem that we in our sitemap include (many) URL's to pages that are marked with noindex? 3. Are there any problems with having a sitemap that includes pages that includes canonical URL's to other pages? Thanks in advance!
Algorithm Updates | | TradingFloor.com0 -
Page details in Google Search
I noticed this morning a drop in the SERPs for a couple of my main keywords. And even though this is a little annoying the more pressing matter is that Google is not displaying the meta title I have specified for the majority of my sites pages, despite one being specified and knowing my site has them in place. Could this sudden change to not using my specified title be the cause of the drop, and why would they be being displayed by Google in the first place, when they are there to be used. The title currently being displayed inthe SERPs is not anything that has been specified in the past or from the previous latest crawl etc. Any insight would be appreciated. Tim
Algorithm Updates | | TimHolmes0 -
Should social widgets be the kind that shares/likes a page, or the kind that adds followers to a brand social page?
I'm wondering if the social widgets on my blog should create a share/like referencing the page or should the social widget create a follower to my brands page on a particular social network? Any ideas?
Algorithm Updates | | salesduke0 -
Webmaster Tools Crawl Stats Drop - Ranks Dropped too
Hello, So I have a situation I haven't seen that happened today. I checked in on my keyword rankings, and all seemed to drop off the rankings in one day. Even googling my domain name, I'm not the first result (when I always have been with multiple sublinks under). There were no unusual errors in the webmaster tools, no issues "Fetching as Google". The only thing I can find is in Crawl Stats, my "Kilobytes downloaded per day" and "Time spent downloading a page (in milliseconds)" dropped a lot in the last day or so (by about 90%). Has anyone had this happen? Any ideas for how to bounce back from this? Domain referenced is mediamegamall.com Thanks!
Algorithm Updates | | Ask_MMM0 -
Reason for massive dropp in targeted keywords?
Our site at www.total-displays.com has ranked well for a variety of keywords over a long period of time. After the change on Jan 18th we have seen a dramatic drop in organic rank across a variety of keywords. Our site has very good unique content and has a large number of pages (it's a Magento site). Advice we have taken is that the site will bounce back after a few weeks but this does not seem to be happening. Any advice please? Thanks Nick
Algorithm Updates | | Total_Displays0 -
Lesser visited, but highly ranked landing paged dropped in rank on Google. Time for a content update?
I noticed that my page one ranked landing pages that don't get a lot of love from me have dropped in rank big time on Google this week. This is a site that has static (meaning, I can't freshen up the content easily) landing pages for products that we sell. The pages that dropped are the ones that have the fewest inbound links, and don't get much attention on the social media side. Our most important landing pages have also dropped, but just a few spots on page one. This is a first for me. Does anyone think that this is a "lack of freshness" penalty? We are still number one on page one for our brand search terms. Would fresh content give me a shot at getting the pages back up? I'm willing to update them slowly, but before I go crazy, I'm reaching out to the pros here.
Algorithm Updates | | Ticket_King0