Reasons for a sharp decline in pages crawled
-
Hello!
I have a site I've been tracking using Moz since July. The site is mainly stagnant with some on page content updates. Starting the first week of December, Moz crawler diagnostics showed that the number of pages crawled decreased from 300 to 100 in a week.
So did the number of errors through. So crawler issues went from 275 to 50 and total pages crawled went from 190 to 125 in a week and this number has stayed the same for the last 5 weeks.
Are the drops a red flag? Or is it ok since errors decreased also? Has anyone else experienced this and found an issue?
FYI: sitemap exists and is submitted via webmaster tools. GWT shows no crawler errors nor blocked URLs.
-
Google is indexing just over 80 URLs, although about 40% of them are developer test URLs (they lead to live pages of the site though). Nothing in robots.txt. No errors.
The Google bot is still crawling, but it's crawling half the pages. What would make it decrease in page crawls? I'm working if there is a broken link or something on the home page that's pointing away from the site... although it's unlikely, I'll check....
-
If you fixed a problem, such as duplicate content, that would mean that we're showing fewer errors and crawling fewer pages, since that problem is fixed. Might that be the case?
-
How many URLs are indexed in Google if you use site:yourdomain.com Has that figure dropped too?
Have you got anything in your robots.txt that could be blocking?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens when most of the website visitors end up at an "noindex" log-in page?
Hi all, As most of the users are visiting our website for log-in, we are planning to deindex login page. As they cannpt find it on SERP, they gonna visit our website and login; I just wonder what happens when most of the visitors just end up at homepage by browsing into an "noindex" page. Obviously it increases bounce rate and exit rate as they just gonna disappear. Is this going to push down us in rankings? What are the other concerns to check about? Thanks
Algorithm Updates | | vtmoz0 -
Log-in page ranking instead of homepage due to high traffic on login page! How to avoid?
Hi all, Our log-in page is ranking in SERP instead of homepage and some times both pages rank for the primary keyword we targeted. We have even dropped. I am looking for a solution for this. Three points here to consider is: Our log-in page is the most visited page and landing page on the website. Even there is the primary keyword in this page or not; same scenario continues Log-in page is the first link bots touch when they crawling any page of our website as log-in page is linked on top navigation menu If we move login page to sub-domain, will it works? I am worrying that we loose so much traffic to our website which will be taken away from log-in page sub domain Please guide with your valuable suggestions. Thanks
Algorithm Updates | | vtmoz0 -
Is it stil a rule that Google will only index pages up to three tiers deep? Or has this changed?
I haven't looked into this in a while, it used to be that you didn't want to bury pages beyond three clicks from the main page. What is the rule now in order to have deep pages indexed?
Algorithm Updates | | seoessentials0 -
Page details in Google Search
I noticed this morning a drop in the SERPs for a couple of my main keywords. And even though this is a little annoying the more pressing matter is that Google is not displaying the meta title I have specified for the majority of my sites pages, despite one being specified and knowing my site has them in place. Could this sudden change to not using my specified title be the cause of the drop, and why would they be being displayed by Google in the first place, when they are there to be used. The title currently being displayed inthe SERPs is not anything that has been specified in the past or from the previous latest crawl etc. Any insight would be appreciated. Tim
Algorithm Updates | | TimHolmes0 -
Do scraped or borrowed articles with my links still pass page rank?
I wrote some articles for Ezine Articles a few years back and i still see links in the ose to my site that are from these articles that were borrowed from the Ezine Articles bank. Do the links in these articles still count toward my site including link juice and anchor text or does google discount them as duplicate content? I was told that Google counts these links for about 3 weeks and then discounts them as duplicate content so it's like they don't exist. Any truth to this or should i make the articles on my site available for people to copy and paste into their blogs as long as they keep my links intact? Thanks, Ron
Algorithm Updates | | Ron100 -
Why is my domain authority (and page authority) plummeting?
In June our domain authority was at a 41. In July we were 38 and ever since then our domain authority is gradually getting worse and worse. We went from a 33 to a 29 in one week! Possible explanations include: Maybe the SEO we hired (for a few months in late 2011) added our domain to some less-than-awesome directories The 301 redirects on our home page are hurting us somehow Duplicate content for URL's with different capitalization (IE: /pages/aboutus and /Pages/AboutUs) Can someone please point me in the right direction? Which of the above possibilities would likely impact domain/page authority? Any other ideas as to why this might be happening? Any suggestions for improving our domain or page authority? Thanks for the help!
Algorithm Updates | | MichaelBrown550 -
Anybody else noticing some serious dung ranking well for no reason lately?
I mean, like the complete opposite of what Panda is supposed to be doing... We've just had a discussion here and each went through a bunch of examples of sites that have sprung up and are ranking with the worst, spammiest backlink profiles ever, and poor content with little to zero on-page SEO. It's not accross the board by any means, and certainly there is no obvious pattern but there were a fair few examples we just looked at (I best not provide them for ethical reasons). It just "feels" like there's something going on...
Algorithm Updates | | SteveOllington0 -
Will google punish us for using formulaic keyword-rich content on different pages on our site?
We have 100 to 150 words of SEO text per page on www.storitz.com. Our challenge is that we are a storage property aggregator with hundreds of metros. We have to distinguish each city with relevant and umique text. If we use a modular approach where we mix and match pre-written (by us) content, demographic and location oriented text in an attempt to create relevant and unique text for multiple (hundreds) of pages on our site, will we be devalued by Google?
Algorithm Updates | | Storitz0