Reasons for a sharp decline in pages crawled
-
Hello!
I have a site I've been tracking using Moz since July. The site is mainly stagnant with some on page content updates. Starting the first week of December, Moz crawler diagnostics showed that the number of pages crawled decreased from 300 to 100 in a week.
So did the number of errors through. So crawler issues went from 275 to 50 and total pages crawled went from 190 to 125 in a week and this number has stayed the same for the last 5 weeks.
Are the drops a red flag? Or is it ok since errors decreased also? Has anyone else experienced this and found an issue?
FYI: sitemap exists and is submitted via webmaster tools. GWT shows no crawler errors nor blocked URLs.
-
Google is indexing just over 80 URLs, although about 40% of them are developer test URLs (they lead to live pages of the site though). Nothing in robots.txt. No errors.
The Google bot is still crawling, but it's crawling half the pages. What would make it decrease in page crawls? I'm working if there is a broken link or something on the home page that's pointing away from the site... although it's unlikely, I'll check....
-
If you fixed a problem, such as duplicate content, that would mean that we're showing fewer errors and crawling fewer pages, since that problem is fixed. Might that be the case?
-
How many URLs are indexed in Google if you use site:yourdomain.com Has that figure dropped too?
Have you got anything in your robots.txt that could be blocking?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console Not Indexing Pages
Hi there! I have a problem that I was hoping someone could help me with. On google search console, my website does not seem to be indexed well. In fact, even after rectifying problems that Moz's on-demand crawl has pointed out, it still does not become "valid". There are some of the excluded pages that Google has pointed out. I have rectified some of the issues but it doesn't seem to be helping. However, when I submitted the sitemap, it says that the URLs were discoverable, hence I am not sure why they can be discovered but are not deemed "valid". I would sincerely appreciate any suggestions or insights as to how can I go about to solve this issue. Thanks! Screenshot+%28341%29.png Screenshot+%28342%29.png Screenshot+%28343%29.png
Algorithm Updates | | Chowsey0 -
Lost Wikipedia page and dropped heavily in rankings. How many of you aware of and experienced this?
Hi all, We lost of our Wikipedia page for 2nd time and we dropped in rankings 2nd time too. I got confused first time whether Wikipedia was the actual reason as we had couple of major changes in our website. But recently it's been clear that losing Wikipedia page is the culprit as we have no website changes around these days. How many of you aware of this and experienced this? Please share your views. Hope this info will help you. Thanks
Algorithm Updates | | vtmoz0 -
Landing page redirect along with complete content
Hi Moz community, We have a page with "keyword" we are targeting in slug like website.com/keyword/. This page doesn't have much back-links or visits like homepage. So we decided to redirect homepage to /keyword page along with complete content. Will this going to hurt? Only change anybody can notice is URL. Are there any risks involved. I think this is the best way to highlight the page we been thinking about. Thanks
Algorithm Updates | | vtmoz0 -
Which is the best way - to have all FAQ pages at one place, or splitted in different sections of the website?
Hi all, We have a lot of FAQ sections on our website, splitted in different places, depending on products, technologies, etc. If we want to optimize our content for Google's Featured Snippets, Voice Search and etc. - what is the best option: to combine them all in one FAQ section? or it doesn't matter for Google that this type of content is not in one place? Thank you!
Algorithm Updates | | lgrozeva0 -
How is this possible? #2 ranking with NO on-page keywords, no backlinks, no sitemap...
Hi everybody. I have a question ... I'm totally stumped. This question is being asked today (November 16th, 2015) just after Google updated something in their algorithm. Nobody seems to know what they did. and it has something to do with the new "Rank Brain" system they're now using. My niche is Logo Design Software (https://www.thelogocreator.com). I had the keywords "logo creator" on the page roughly 7 times. After Google updated, I lost about 10 spots and as of this writing, I've dropped to #15. So, maybe I over optimized. fine. Noticing that for the keyword "logo creator" ... NONE of the top 14 spots actually have "logo creator" in their page title and NONE of them have more that 2 instances (if any) of the keyword "logo creator" on the actual page. So I removed ALL instances of my keyword "logo creator" from my home page - used the Webmaster's Fetch Tool and moved up a few spots instantly. So what the heck? And the #2 spot for that keyword is www.logomakr.com - they have NO words at all on their pages, no blog, no sitemap and far fewer links than anybody in the top 10. Can anybody reading this shed some light? Marc Marc Sylvester
Algorithm Updates | | Laughingbird
Laughingbird Software0 -
MOZ.com Page Rank of 2?
I don't recall the page rank of SEOMoz.com prior to the company's change to MOZ.com. But did notice that MOZ.com currently has a Page Rank of 2 (which I find weird since it's such a strong, content rich, highly-regarded site). I'd be interested in hearing about findings from the MOZ.com team on why the low PR and how has it affected your site since the change? (...and perhaps a look at the future through a crystal ball 🙂 I recall reading the MOZ domain changing article titled "Domain Migrations: Surviving the "Perfect Storm" of Site Changes" which had great info and addresses some reasons for PR loss in the 'Traffic and Ranking Loss' section: http://moz.com/blog/domain-migration-lessons
Algorithm Updates | | Prospector-Plastics0 -
When was the last algorithm update? One of my pages has dropped significantly this week
One of my pages dropped 22 places last week and I'm not sure why - can any body give me some suggestions to why this might have happened?
Algorithm Updates | | lindsayjhopkins0 -
Google showing different pages for same search term in uk and usa
Hi Guys, I have an interesting question and think Google is being a bit strange.. Can anyone tell me why when I input the term design agency in Google.co.uk it shows one page, but when i tyupe in the same search term in Google.com (worldwide search) it shows another page.. Any ideas guys? Is this not bit strange?? Any help here be much appreciated.. Thanks Gareth
Algorithm Updates | | GAZ090