Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to Diagnose "Crawled - Currently Not Indexed" in Google Search Console
The new Google Search Console gives a ton of information about which pages were excluded and why, but one that I'm struggling with is "crawled - currently not indexed". I have some clients that have fallen into this pit and I've identified one reason why it's occurring on some of them - they have multiple websites covering the same information (local businesses) - but others I'm completely flummoxed. Does anyone have any experience figuring this one out?
Reporting & Analytics | | brettmandoes2 -
Bounce Rate: what is it EXACTLY?
Hi everyone: we all know the term 'Bounce Rate'. I'd like to think i have a good idea of what BR is....but some things are not really clear to me. Time to call in the experts. Question #1: What EXACTLY will stop Google from considering the visit as a bounce? As discussed not too long ago in this topic https://moz.com/community/q/will-this-fix-my-bounce-rate
Reporting & Analytics | | BasKierkels
Ruben wrote: "..what it basically means is that someone clicks on your SERP, and then clicks back to google? But, it doesn't matter if they spent 10 minutes on your page or 10 seconds" Jessica Conflitti wrote a reply in which she basically said that it might be a good idea to have visitors click to a different page OR a PDF-file. That's where my confusion has been for some time now: Clicking on a PDF-document, an image in the page that opens with Fancybox, a link to a different domain? Or can it only be a different URL on the same domain? The way i would expect it to be:
Pages contain the GA-tracking code. So am i right by thinking that Google needs to have the same GA-tracking code to be loaded twice? Because only at that point will they have two datapoints. And only then will they be able to tell that the visitor hasn't left. By clicking a PDF-document - as described by Jessica - you wouldn't load the GA-code twice. So I would expect that clicking a PDF does not make a difference for the BR. Don't get me wrong: i like the article but it is this detail that throws me off. IF Google can read or capture these clicks, what other elements can be used to reduce bounce rate? Clicking on a YouTube-video embedded in the page? I'm asking this because i want to get this right. Question #2: how much weight does BR have on Time on Page, Engagement, etc? We know Google is taking a lot of things into consideration when calculating the value of a URL or domain. So how much should we care for BR if we know the Time on Page is good and a large percentage of people are frequently returning? How about your experiences or knowledge on that? Really looking forward to your replies and help on clearing this topic for me. And perhaps some other readers as well! Bas0 -
Main Website Redirects to Mobile Website, Mobile Website counts this as direct traffic, is there a way to tell what the source/medium is?
Hello, The situation is that someone is arriving on my main website https://www.example.com and being redirected to http://m.example.com. When this happens my analytics says that the traffic is all direct coming to my mobile site. However, I know people clicking on my google cpc, and some google organic users are hitting the main website and being redirected. Before we didn't have as good of a redirect on our main website so I could tell organic and cpc traffic coming in, now my main website has a huge drop in these categories because they are redirecting to mobile but I can't tell on my mobile how much traffic from each is going to the mobile site. Is there a way to fix this? Is it because my main website is https:// and mobile is a http:// (as I know that sometimes makes traffic direct) or is it a bigger problem that can't be resolved? Thanks
Reporting & Analytics | | oxfordseminars0 -
My GWT tells me that verification has failed numerous occasions - will this stop my site being crawled?
I launched www.over50choices.co.uk 6 weeks ago and have had trouble with google indexing and crawling all pages. It tells me 143 submitted & 129 Indexed, but the site has 166 pages? It still shows the old home page image in GWT - which is v annoying! Whilst the site is verified by GA & HTML Tag, it tells me in the Verification section that "reverification failed" on numerous occasions - they seem correspond with when google trys to process the site map. Is this a coincidence ie verification fails when its trying to process the site map, which in turn is leaving me with an out of date site map and therefore not all my pages submitted or crawled? Or will this not effect the googles ability to crawl the site? Your help please. Ash
Reporting & Analytics | | AshShep10 -
How to track and verify 301 redirects from 22 old websites to a new one?
Me and my team have just finished a website which will replace 22 old ones (6 main domains and 16 sub-domains, 19 in wordpress and 3 in Joomla). This means more than 20,000 links that we're trying to redirect with the most granularity possible. But we're experiencing a lot of conflicts because of the quantity and variety of links and the interference of Wordpress. And I'm having trouble finding which links are working and which aren't. I have a URL list in a plain .txt file for every website, is there a tool that can check these links for me? I've tested many wordpress plugins, but I didn't find a good one. Thanks!
Reporting & Analytics | | bernardovailati0 -
Why is a section of our website dropping in&out of Google SERPs?
In July 2011 we started a news section that has it's own 'subfolder' /news/ (http://www.chorder.com/news/new_gear/, http://www.chorder.com/news/gear_deals/ etc.) The whole news section is dropping in&out of Google SERP's since late October, as show in attached graph. All news texts are real deal, written by our own staff, linked from homepage. Any idea why this happens and how to prevent it? cmqky.png
Reporting & Analytics | | imventurer0 -
Bounce rates plummeted
In Google Analytics, my average bounce rates plummeted basically overnight. I went from a consistent average daily bounce rate of about 65% to an average daily bounce rate near 5%. My average number of visitors has stayed the same. I don't think there is any significant change I made to my site that may have caused this. Has anyone else had the same problem and know why it happened and how to fix it? Thanks in advance!
Reporting & Analytics | | Ericc220 -
How do you add Facebook Insights to Multiple Websites?
If I have 4 top-level domain websites all for the same corporate company but with different focuses (ie/ engineering, technology etc.) and I want to add Facebook insights, should I add the same meta tag to all websites or should I create separate meta tags? If I create seperate ones, the issue is that we have one main facebook page that has multiple admins that I would like to give access to but it looks like you can only associate one website to one facebook page. So from what I can tell, if I were to create separate ones, I would have to setup fake facebook pages or fake profiles in order to get different facebook insights meta tags? I'm hoping there is a better way... or maybe I should just put the same meta tag code on all 4 websites? Thanks for your help!
Reporting & Analytics | | randstadsocial0