What would cause a drastic drop in pages crawled per day?
-
The site didn't go down.
There were no drop in rankings, or traffic.
But we went from averaging 150,000 pages crawled per day, to ~1000 pages crawled per day.
We're now back up to ~100,000 crawled per day, but we went more than a week with only 1000 pages being crawled daily.
The question is, what could cause this drastic (but temporary) reduction in pages crawled?
-
I wish that were the case, but the site wasn't down.
I looked into the errors, they were redirecting to a subdomain that no longer exists.
-
So several times in one month the entire site couldn't be reached. That's pretty significant. Personally I don't have any clients with that many down-times so can only assume that's the cause or at least a partial cause. And more important, a red flag that would prompt me to find a better hosting provider if it were my site.
-
The drop happened March 28th.
There was a "domain name not found" on march 30th (two more on the 22nd, 18th, 12th, and 10th)
-
There could be several factors. When did it occur? Did you see any other crawl errors reported? And unfortunately, the other unknown comes from the fact that Google's own system is both far from perfect and sometimes crawl volume is affected by their own system.
Unless I see crawl errors or an increase in pages not found during or leading up to that period, or more important, see a corresponding significant drop in organic traffic, personally I just chalk it up to the complexity of the web.
-
Hi Alan!
There were no spikes in kb per day or time spent downloading a page.
-
Fatwallet
Have you checked Google Webmaster Tools for crawl errors and other metrics? I had a client recently who had a severe slowdown in their server network which showed up on page crawl speed time as a huge spike - pages loading five times slower than normal. They subsequently had a dip in pages crawled due to the bottleneck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Magento 1.9 SEO. I have product pages with identical On Page SEO score in the 90's. Some pull up Google page 1 some won't pull up at all. I am searching for the exact title on that page.
I have a website built on Magento 1.9. There are approximately 290,000 part numbers on the site. I am sampling Google SERP results. About 20% of the keywords show up on page 1 position 5 thru 10. 80% don't show up at all. When I do a MOZ page score I get high 80's to 90's. A page score of 89 on one part # may show up on page one, An identical page score on a different part # can't be found on Google. I am searching for the exact part # in the page title. Any thoughts on what may be going on? This seems to me like a Magento SEO issue.
Intermediate & Advanced SEO | | CTOPDS0 -
Too many on page links
Hi I know previously it was recommended to stick to under 100 links on the page, but I've run a crawl and mine are over this now with 130+ How important is this now? I've read a few articles to say it's not as crucial as before. Thanks!
Intermediate & Advanced SEO | | BeckyKey1 -
My page ranking dropped after I made it responsive
Recently, I made one of my site page responsive. It was ranking on 13th Position and after I made the page responsive, it dropped to 24 desktop and 35 on mobile rankings. I ran mobile friendly test and it said "awesome, your page is mobile friendly". Not sure, what went wrong here. Page in question is: https://www.itcontractorsuk.com/ Can someone please advice? Please Note: My site is partially responsive.
Intermediate & Advanced SEO | | ThinkWebUK0 -
Ranking drop
Hello there, Based on Moz's rank tracker our keywords ranking have been dropping in ranking. Does anyone know what might be the cause? We have been building quality "white hat" links which are very relevant to our niche Thanks, Robert
Intermediate & Advanced SEO | | roberthseo0 -
On Page vs Off Page - Which Has a Greater Effect on Rankings?
Hi Mozzers, My site will be migrating to a new domain soon, and I am not sure how to spend my time. Should I be optimizing our content for keywords, improving internal linking, and writing new content - or should I be doing link building for our current domain (or the new one)? Is there a certain ratio that determines rankings which can help me prioritize these to-dos?, such as 70:30 in favor of link-building? Thanks for any help you can offer!
Intermediate & Advanced SEO | | Travis-W0 -
Rankings Dropped
Hello, I wonder if anyone can point us in the right direction. Our main domain name has got a good SEO profile the domain is 12 years old and we have some good reputable links. On the 7th of Jan our website dropped in the rankings from 3rd to 10th for one of our main keywords and from then onwards some of our other keywords have dropped. We have a lot of landing pages that target specific keywords which look like a template page but just with the content changed. Can anyone pin point what could of caused this problem or has anyone experienced this before and knows how to fix it. I personally think we have been hit by the panda. Thanks, Scott
Intermediate & Advanced SEO | | ScottBaxterWW0 -
How do I fix the error duplicate page content and duplicate page title?
On my site www.millsheating.co.uk I have the error message as per the question title. The conflict is coming from these two pages which are effectively the same page: www.millsheating.co.uk www.millsheating.co.uk/index I have added a htaccess file to the root folder as I thought (hoped) it would fix the problem but I doesn't appear to have done so. this is the content of the htaccess file: Options +FollowSymLinks RewriteEngine On RewriteCond %{HTTP_HOST} ^millsheating.co.uk RewriteRule (.*) http://www.millsheating.co.uk/$1 [R=301,L] RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index\.html\ HTTP/ RewriteRule ^index\.html$ http://www.millsheating.co.uk/ [R=301,L] AddType x-mapp-php5 .php
Intermediate & Advanced SEO | | JasonHegarty0 -
What to call pages
I reckon I've bagged one of the most interesting SEO projects of the year. My new client is selling vibrators. The site is not even in development yet but they want to make it fun and friendly and take away the stigma and "seediness" of the product. Anyway, the owenr has presented a list of "places" within this site which are places where the products are going to be showcased. These are along the lines of, Royal Rabbits Palace, Clitoral Courtyard, Dungeon Dildos, Magical G-arden etc. (there is a bit shreky/fariy tale thing going on) Clearly, these places add a lot to the look and feel of the site but as URL's and Titles, they are clearly not optimal in an SEO sense. What is for the best...making sure we shift the owner back into SEO best practice or hope that having these weird and wonderful names for the pages is going to add enough to the user experience to make it worthwhile to let through. FYI, did you know you can get vibrators that you can plug an ipod into. Man, I've seen some weird things researching this client!
Intermediate & Advanced SEO | | FDC0