Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
If website users don't accept GDPR cookie consent, does that prevent GA-GTM from tracking pageviews and any traffic from that user that would cause significant traffic decreases?
I've been doing a lot research on GDPR impact and implementation with GTM-GA for clients, but it's been 12 months since GDPR has gone live I haven't found anything on how GA traffic has been impacted if users don't accept cookie consent. However, I'm personally seeing GA accounts taking huge losses in traffic since implementing GDPR cookie solutions (because GTM/GA tags aren't firing until cookies are accepted). Is it common for websites to see significant decreases in traffic due to too many users not accepting cookie consent? Are there alternative solutions to avoid traffic loss like that and still maintain GDPR compliance? It seems to me that the industry underestimated how many people won't accept cookie consent. Most of the documentation and articles around GDPR's start (May 2018) didn't foresee or cover that aspect properly, everything seems to be technically focused with the assumption that if implemented properly most people would accept cookie consent, but I'm personally not seeing that trend and it's destroying GA data (lost traffic, minimal source attribution, inaccurate behavior data, etc). Thanks.
Reporting & Analytics | | Kickboard2 -
Need your Opinion on Bounce Rate Analysis
I'm currently doing a bounce rate analysis for our resource pages. These are information article pages - mix of plain texts and those containing either images, infographics, videos or even podcasts. By the way, I did search for bounce rate topics here, but I felt like i still need to post this. Unless I've overlooked a similar post, my apologies. It's a first for me to do an in-depth BR analysis, so I need to clarify few things. What is a good or bad range bounce rate? Is there even a range comparison? Like when can you say a bounce rate is high for an information type page? I've read some stuff online but they're confusing. What other Analytics factors should I consider looking at together with bounce rate? For pages (which purposely educate visitors) with high bounce rate, can you guys suggest tips to improve it? I would appreciate and value any advise. Thanks a lot!
Reporting & Analytics | | ktrich1 -
Angular website and ranking
Hi guys Unfortunately I have to optimize the angular website, but I don't know how google see my website. Seo quacke (seo extension) doesn't get data from this website: https://cafegardesh.com and sitemap generation tools just crawl 1 page of this website. why? How find that google really crawl and index angular website?
Reporting & Analytics | | denakalami0 -
Is there an easy way to switch hundreds of websites to https in GSC?
My company has hundreds of websites setup in Google Search Console but will soon be moving them all to secure domains. Is there an easy way to make the switch in GSC or do we have to change the address one by one?
Reporting & Analytics | | MJTrevens0 -
How do I track specific referral traffics journey through a website?
Hello, A client has asked us to track the journey each separate referral traffic visitor takes through out the website. I have had a look through analytics and am not sure how to ensure I can do this for all referral traffic visitors? Can anyone help? Thank you.
Reporting & Analytics | | mblsolutions0 -
Tracking time spent on a section of a website in Google Analytics
Hi, I've been asked by a client to track time spent or number of pages visited on a specific section of their website using Google Analytics but can't see how to do this. For example, they have a "golf" section within their site and want to measure how many people either visit 5 page or more within the golf section or spend at least 6 minutes browsing the various golf section pages. Can anyone advise how if this can be done, and if so, how I go about it. Thanks
Reporting & Analytics | | geckonm0 -
What about this (google crawl)?
Recently we did a serious effort on SEO with SEO Yoast (Wordpress). And after a few months of tweaking old articles we get this impact on crawl search.. Is this graph normal? s1TQgv9.png
Reporting & Analytics | | noodweerbenelux0 -
SEOMoz & Google Webmaster Tools crawl error conflicting info
Site im working on has zero crawl errors according to SEOMoz (it did previously have lots since ironed out) but now looking at GWebmaster Tools saying 5000 errors. Date of those are not that recent but Webmaster Tools line graph of errors still showing aprox 5000 up to yesterday There is an option to bulk action/tick them all as fixed so thinking/hoping GWT just keeping a historical record that can now be deleted since no longer applicable. However i'm not confident this is the case since still showing on the line graph. Any ideas re this anomalous info (can i delete and forget in GWT) ? Also side question I take it its not possible to link a GA property with a GWT account if created with different logins/accounts ? Many Thanks Dan
Reporting & Analytics | | Dan-Lawrence0