2 days in the past week Google has crawled 10x the average pages crawled per day. What does this mean?
-
For the past 3 months my site www.dlawlesshardware.com has had an average of about 400 pages crawled per day by google. We have just over 6,000 indexed pages.
However, twice in the last week, Google crawled an enormous percentage of my site. After averaging 400 pages crawled for the last 3 months, the last 4 days of crawl stats say the following.
2/1 - 4,373 pages crawled
2/2 - 367 pages crawled
2/3 - 4,777 pages crawled
2/4 - 437 pages crawled
What is the deal with these enormous spike in pages crawled per day? Of course, there are also corresponding spikes in kilobytes downloaded per day.
Essentially, Google averages crawling about 6% of my site a day. But twice in the last week, Google decided to crawl just under 80% of my site.
Has this happened to anyone else? Any ideas? I have literally no idea what this means and I haven't found anyone else with the same problem. Only people complaining about massive DROPS in pages crawled per day.
Here is a screenshot from Webmaster Tools: http://imgur.com/kpnQ8EP
The drop in time spent downloading a page corresponded exactly to an improvement in our CSS. So that probably doesn't need to be considered, although I'm up for any theories from anyone about anything.
-
2 weeks later, 2 more large spikes of 10X average...nothing bad is happening...oh well...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do modal pop-ups impact the Google Analytics of the host page?
We have an instance of a page where visitors can click a button to start an interactive quiz. The quiz pops up in a modal window that references another domain (the interactive content provider). Will the person completing the quiz in the modal pop-up still be counted as an active visitor on the original host page during the time they are completing the quiz?
Reporting & Analytics | | MuhammadInc0 -
Identifying Bots in Google Analytics
Hi there, While you can now filter out bots and spiders in Google Analytics, I'm interested in how you identify a bots and spiders in the first place. For example, it used to be thought that Googlebot wouldn't appear in GA as it 'couldn't process Javascript' but now Google has announced new developments for its crawler with regards to interpreting javascript and CSS, this argument isn't as cut and dry. I'm not suggesting Googlebot appears in Google Analytics, but I am saying that you can't make the case that it won't appear only because it can't interpret JavaScript. So, I'm interested to see what metrics you use to identify a bot? For me, the mix of Users > Browser, Users > Operating System Version is still quite handy, but is it possible to identify individual bots and spiders within Google Analytics? And would Googlebot appear?
Reporting & Analytics | | ecommercebc0 -
Moving data between Google Analytics Properties
Last summer we setup another Google Analytics property for us with Universal Analytics and have been running this alongside the old Google Analtyics property. is there a way of exporting all the old data from the old property into the new Universal Analytics property?
Reporting & Analytics | | ese0 -
Google Analytics (not provided)
Hi A greater and greater % of clicks is showing as "not provided" - does any one know of any good external tracking software which will show this information? Thanks
Reporting & Analytics | | jj34340 -
Setting up Google Analytics for Subsites
I currently have one main .com site and am planning on launching geo-location subsites .co.uk, .com.au, .ru, etc... Traffic will flow between both sites and some of the content on the subsites will be duplicate and therefore include a canonical tag to the main site. I want to set up GA to capture who is going to the subsites and vice versa and correctly capture crossover traffic. Any advice on implementing advanced analytics directly (or links to sources that will direct me the right direction for this project)
Reporting & Analytics | | theLotter0 -
Where are google analytics stats for iphone4
hi We were looking at the Google Analytics for one of our sites and noticed that there were NO pageviews from device=iphone and resolution=640x960 in the report. Given that iphone4 is supposed to be 640x960, and would be the most popular device (at least in our offices and everyone I know), it seems wierd. I sorted the Mobile Devices report by device and resolution to see what was available. The first 160 results were all device=not set. Finally got to device=iPhone and there were three entries: resolution 0x0 had 11 views resolution 320x396 had 45 views resolution 320x480 had 3,944 views. Hopefully all iphone4 users havent been classified as not set. Or is it possible that iphone4s claim to be 320x480 in browsers, as per http://www.alistapart.com/articles/a-pixel-identity-crisis/ Even worse, if I look at the Samsung Galaxy S II (myown phone), there are over 30 screen resolution combinations. Does anyone have anything to shed on this? I asked about it on the google analytics twitter account last week but havent had a response. Are there other analytics solutions that would distinguish between the iphones? Warning - this is a link to a large image, with the not set stats at the top. 6Sjji
Reporting & Analytics | | ozgeekmum0 -
In-Page Analytics
Hey folks, In the in-page analytics section of GA I am wanting to get the % of clicks on a particular link over a large selection of dates. Does anyone know how to pull the data without having to do it day by day? Ideally I want to pull it all at once. There might of course be other ways to do this, if so please let me know. Thanks for your help. Cheers,
Reporting & Analytics | | CraigAddyman0 -
Google Analytics | REAL TIME
So I noticed today that there is now Real Time Data: http://analytics.blogspot.com/2011/09/whats-happening-on-your-site-right-now.html and I cannot figure out how to access this.
Reporting & Analytics | | joseph.chambers1