2 days in the past week Google has crawled 10x the average pages crawled per day. What does this mean?
-
For the past 3 months my site www.dlawlesshardware.com has had an average of about 400 pages crawled per day by google. We have just over 6,000 indexed pages.
However, twice in the last week, Google crawled an enormous percentage of my site. After averaging 400 pages crawled for the last 3 months, the last 4 days of crawl stats say the following.
2/1 - 4,373 pages crawled
2/2 - 367 pages crawled
2/3 - 4,777 pages crawled
2/4 - 437 pages crawled
What is the deal with these enormous spike in pages crawled per day? Of course, there are also corresponding spikes in kilobytes downloaded per day.
Essentially, Google averages crawling about 6% of my site a day. But twice in the last week, Google decided to crawl just under 80% of my site.
Has this happened to anyone else? Any ideas? I have literally no idea what this means and I haven't found anyone else with the same problem. Only people complaining about massive DROPS in pages crawled per day.
Here is a screenshot from Webmaster Tools: http://imgur.com/kpnQ8EP
The drop in time spent downloading a page corresponded exactly to an improvement in our CSS. So that probably doesn't need to be considered, although I'm up for any theories from anyone about anything.
-
2 weeks later, 2 more large spikes of 10X average...nothing bad is happening...oh well...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Was Google Analytics and Adsense Down Today?
For the last 4 hours of so we were registering zero users and Adsense reporting has not changed. We checked the site, and there were no problems. It seems for some reason there was no reporting. Just now it came back up and we are showing live traffic. Trying to figure out if this was a problem specific to us or if it is on Google's end. Thanks,
Reporting & Analytics | | akin670 -
Variables in Google Tag Manager
Last week I got a great answer here on how to implement GTM and cross domain tracking, now that we have that in place I'm looking for some more advice 😉 I'm wondering that if we push variabels into the data-layer of GTM, can we actually use those variables in reports as well? Do they get recorded in Google Analytics? I'd like to use some data that we push to the data layer for segmentation purposes. Anybody an idea how to achieve this?
Reporting & Analytics | | jorisbrabants0 -
Google Analytics View Filters
Using the same GA property, I would like to set up three filtered views: 1. Tracking across one subdomain and one primary domain (example: shop.example.com & example.com) 2. Track only primary domain (example.com) 3. Track only subdomain (shop.example.com) Can this be achieved by using view filters? If so, how do they need to be set? Also, according to this article: https://moz.com/blog/cross-domain-subdomain-tracking-in-google-analytics, with cross domain tracking, I need to ignore self-referrals, which can only be done at the property level. If set up to ignore example.com referrals, will this cause problems with filter 2 and 3?
Reporting & Analytics | | Evan340 -
Moz Crawl shows over 100 times more pages than my site has?
The latest crawl stats are attached. My site has just over 300 pages? Wondering what I have done wrong? RRv3fR0
Reporting & Analytics | | Billboard20120 -
Is Google still differentiating TLD?
I need to find an article or something with factual findings that shows Google no longer gives extra value to .com, .net, .us, edu. etc. or proving that is still does.
Reporting & Analytics | | PPI0 -
Tips for migrating Google News traffic?
We are about to relaunch a site that gets a lot of Google News traffic. We are not changing domain, but the site structure is changing greatly, with the URLs of both news index pages and articles being shaken up. Obviously, we've 301-ed every page to its closest equivalent on the new site. We've also got a news sitemap. As we are not changing domain, is there anything further we need to do to help protect our Google News traffic. On a related note, does anyone have a relaible way of measuring traffic from Google News listings in universal search?
Reporting & Analytics | | Dennis-529610 -
If a page bounces in the woods, can Google Panda hear it?
I have read that after the Panda update a site's bounce rate is an important ranking metric. However, can anyone confirm whether all pages count equally? For instance, my home page gets 5000% more traffic than Deep Page X. If Deep Page X has a poor bounce rate, does it matter as much as if my Homepage has a bad bounce rate? I am guessing not, but wanted to open it up for discussion. If not, it has me wondering on what to do for some of my database driven content. I have some dynamically created pages that have higher bounce rates and minimal unique content. They aren't pure spam or junk, but are likely only about 1% unique from one another. Sounds like a no brainer change post-Panda, right? Well, what if I was the only one targeting the keywords for these pages? The pages pull from info I stored on the U.S. government stimulus program (related to my industry). It then has just about every city, state and county combo in the country for my product. For instance, a page <title>might be "Flemington, NJ Widgets - Somerset County". Something that no one else is targeting and drives minimal traffic.</p> <p> </p> <p>Do I take this content down? I didn't have any affects, positive or negative from Panda, so I am hesitant to take down thousands of Google cached pages.</p></title>
Reporting & Analytics | | TheDude0 -
Does Google Analytics use your data against you?
I couldn't find this question answered anywhere in Q&A, so I apologize if it's a duplicate of another post. I heard, about a year ago on either Web Pro World, or Warrior Forum that Google uses your visitor data in your Google Analytic account to rank your site. Someone said that when they took out the Google code, their site went from the third to the first page within 48 hours. That was then verified over the next couple of weeks by others. Their thought was that regardless of the optimized page and incoming link, if the site wasn't getting visitors, then it would be penalized. Since Google has the data, they would be able to use it. I then started using another, paid, solution - getclicky.com. While I like clicky, there is some info Google has that clicky doesn't, everyone integrates with Google analytics - like SEOmoz, and I'm paying a monthly fee. Now that I'm a part of a community of experts, what do you think? Have you noticed Google ranking you based on your analytics data? Has anyone experienced this, or heard about it before? Because I'd like to go back to using Google analytics. Thanks!
Reporting & Analytics | | DallasBonsai0