What would cause a drastic drop in pages crawled per day?
-
The site didn't go down.
There were no drop in rankings, or traffic.
But we went from averaging 150,000 pages crawled per day, to ~1000 pages crawled per day.
We're now back up to ~100,000 crawled per day, but we went more than a week with only 1000 pages being crawled daily.
The question is, what could cause this drastic (but temporary) reduction in pages crawled?
-
I wish that were the case, but the site wasn't down.
I looked into the errors, they were redirecting to a subdomain that no longer exists.
-
So several times in one month the entire site couldn't be reached. That's pretty significant. Personally I don't have any clients with that many down-times so can only assume that's the cause or at least a partial cause. And more important, a red flag that would prompt me to find a better hosting provider if it were my site.
-
The drop happened March 28th.
There was a "domain name not found" on march 30th (two more on the 22nd, 18th, 12th, and 10th)
-
There could be several factors. When did it occur? Did you see any other crawl errors reported? And unfortunately, the other unknown comes from the fact that Google's own system is both far from perfect and sometimes crawl volume is affected by their own system.
Unless I see crawl errors or an increase in pages not found during or leading up to that period, or more important, see a corresponding significant drop in organic traffic, personally I just chalk it up to the complexity of the web.
-
Hi Alan!
There were no spikes in kb per day or time spent downloading a page.
-
Fatwallet
Have you checked Google Webmaster Tools for crawl errors and other metrics? I had a client recently who had a severe slowdown in their server network which showed up on page crawl speed time as a huge spike - pages loading five times slower than normal. They subsequently had a dip in pages crawled due to the bottleneck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pillar pages and blog pages
Hello, I was watching this video about pillar pages https://www.youtube.com/watch?v=Db3TpDZf_to and tried to apply it to my self but find it impossible to do (but maybe I am looking at it the wrong way). Let's say I want to rank on "Normandy bike tou"r. I created a pillar page about "Normandy bike tour" what would be the topics of the subpages boosting that pillar page. I know that it should be questions people have but in the tourism industry they don't have any, they just want us to make them dream !! I though about doing more general blog pages about things such as : Places to rent a bike in Normandy or in XYZ city ? ( related to biking) Or the landing sites in Normandy ? (not related to biking) Is it the way to do it, what do you recommend ? Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Home page vs inner page?
do you believe that the advantage of targeting a search term on the home page is now worse off than before? as I understand it ctr is a big factor now And as far as i can see if two pages are equal on page etc the better ctr will win out, the issue with the home page is the serp stars cannot be used hence the ctr on a product page will be higher? I feel if you where able to get a home page up quicker (1 year instead of two) you still lost out in the end due to the product page winning on ctr? do you think this is correct?
Intermediate & Advanced SEO | | BobAnderson0 -
Google is indexing wrong page for search terms not on that page
I’m having a problem … the wrong page is indexing with Google, for search phrases “not on that page”. Explained … On a website I developed, I have four products. For example sake, we’ll say these four products are: Sneakers (search phrase: sneakers) Boots (search phrase: boots) Sandals (search phrase: sandals) High heels (search phrase: high heels) Error: What is going “wrong” is … When the search phrase “high heels” is indexed by Google, my “Sneakers” page is being indexed instead (and ranking very well, like #2). The page that SHOULD be indexing, is the “High heels” page (not the sneakers page – this is the wrong search phrase, and it’s not even on that product page – not in URL, not in H1 tags, not in title, not in page text – nowhere, except for in the top navigation link). Clue #1 … this same error is ALSO happening for my other search phrases, in exactly the same manner. i.e. … the search phrase “sandals” is ALSO resulting in my “Sneakers” page being indexed, by Google. Clue #2 … this error is NOT happening with Bing (the proper pages are correctly indexing with the proper search phrases, in Bing). Note 1: MOZ has given all my product pages an “A” ranking, for optimization. Note 2: This is a WordPress website. Note 3: I had recently migrated (3 months ago) most of this new website’s page content (but not the “Sneakers” page – this page is new) from an old, existing website (not mine), which had been indexing OK for these search phrases. Note 4: 301 redirects were used, for all of the OLD website pages, to the new website. I have tried everything I can think of to fix this, over a period of more than 30 days. Nothing has worked. I think the “clues” (it indexes properly in Bing) are useful, but I need help. Thoughts?
Intermediate & Advanced SEO | | MG_Lomb_SEO0 -
500 and 508 pages?
Hi we just did a massive deepcrawl (using the tool deepcrawl.co.uk/) on the site: http://tinyurl.com/nu6ww4z http://i.imgur.com/vGmCdHK.jpg Which reported a lot of URLs as either 508 and 500 errors. For the URLs as reported as either 508 or 500 after the deep crawl crawl finished we put them directly into screaming frog and they all came back with status code 200. Could it be because Deep Crawl hammered the site and the server couldn't handle the load or something? Cheers, Chris
Intermediate & Advanced SEO | | jayoliverwright0 -
Crawl budget
I am a believer in this concept, showing google less pages will increase their importance. here is my question: I manage a website with millions of pages, high organic traffic (lower than before). I do believe that too many pages are crawled. there are pages that I do not need google to crawl and followed. noindex follow does not save on the mentioned crawl budget. deleting those pages is not possible. any advice will be appreciated. If I disallow those pages I am missing on pages that help my important pages.
Intermediate & Advanced SEO | | ciznerguy2 -
Our client's web property recently switched over to secure pages (https) however there non secure pages (http) are still being indexed in Google. Should we request in GWMT to have the non secure pages deindexed?
Our client recently switched over to https via new SSL. They have also implemented rel canonicals for most of their internal webpages (that point to the https). However many of their non secure webpages are still being indexed by Google. We have access to their GWMT for both the secure and non secure pages.
Intermediate & Advanced SEO | | RosemaryB
Should we just let Google figure out what to do with the non secure pages? We would like to setup 301 redirects from the old non secure pages to the new secure pages, but were not sure if this is going to happen. We thought about requesting in GWMT for Google to remove the non secure pages. However we felt this was pretty drastic. Any recommendations would be much appreciated.0 -
Product pages content
Hi! I'm doing some SEO work for a new client. I've been tasked with boosting some of their products, such as http://www.lawnmowersdirect.co.uk/product/self-propelled-rear-roller-rotary-petrol-lawnmowers/honda-hrx426qx. It's currently #48 for the term Honda Izy HRG465SD, while http://www.justlawnmowers.co.uk/lawnmowers/honda-izy-hrg-465-sd.htm is #2, behind Amazon. Regarding links, there's no great shakes between the pages or even the domains. However, there's major difference in content. I'm happy to completely revamp it, I just wanted to check I'm not missing anything out before starting to rewrite it altogether! Thanks
Intermediate & Advanced SEO | | neooptic0 -
Dropped 50 places for Page Title Change
Hey Guys, I've got a keyword set that is really annoying me. You see on the 19th July I changed my page title for the home page to what I thought would be something more competitive and instead of climbing from rank 9/10 I dropped out of the top 50. So I waited 2 weeks to see if it changed and it just stayed out of the top 50. After two weeks I changed it back again and I've been out of the top 50 ever since. I havnt made any real significant changes to the site as I've held off waiting for the keyword ranking to return but I dont think it will. I was trying to improve the keyword phrase "bahamas all inclusive" for the website www.smallhope.com. I'd managed to get on to page 1 rank 9/10 depending. I changed the title; from "Bahamas All Inclusive - Andros Bahamas Hotel - Bahamas Vacation - Small Hope Bay Lodge All-Inclusive" to "Bahamas All Inclusive Vacation - Small Hope Bay Lodge All-Inclusive" This is really bad for us and I'm not sure what to do. I had a whole plan in place starting with the Title and moving on to the H1 tags, but now I'm not so sure. I've put everything on hold. If this was you what would you do to get back to page 1? Thanks, Adam
Intermediate & Advanced SEO | | NaescentAdam0