Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help on how to test the DA, PA of the website
Please help me. How to check the DA, PA of the website https://toolaim.com/ . I want to know any quality website to take care of it more. But at present I do not know any forum quality is good and trust. Thank you
Reporting & Analytics | | gogoanimetp0 -
Will adjusted bounce rate affect avg time on page?
I recently read Rob Beirne's piece on how and why to set up an adjusted bounce rate in Google Analytics (https://moz.com/blog/adjusted-bounce-rate). I am getting myself ready to talk to our site team about why we should set up an adjusted bounce rate and am anticipating some questions I'd like to be able to answer: 1. Will an adjusted bounce rate improve the accuracy of our avg time on page metrics? 2. Are we able to keep the unadjusted bounce rate in GA as well, so we can compare the two metrics if we ever need to? Does anyone know the answers to these questions? Any help would be much appreciated!
Reporting & Analytics | | seoisfun0 -
When my new website went live I saw a huge spike in traffic for 3 weeks but then it halved again. Why would this happen?
My new website went live on 13th April and the organic traffic levels increased by 110%, staying at that level for 3 weeks. Then exactly 3 weeks later the organic traffic halved again, going back to the same level as the old site. Has anyone seen this happen before or worked out why?! Thanks Viki P2R76O9
Reporting & Analytics | | TWB0 -
Is there an easy way to switch hundreds of websites to https in GSC?
My company has hundreds of websites setup in Google Search Console but will soon be moving them all to secure domains. Is there an easy way to make the switch in GSC or do we have to change the address one by one?
Reporting & Analytics | | MJTrevens0 -
Is there an automated way to determine which pages of your website are getting 0 traffic?
I'm doing a content audit on my company website and want to identify pages with zero traffic. I can use GA for low traffic, but not zero traffic. I can do this manually, but it would take a long time. Are there any tools to help me determine these pages?
Reporting & Analytics | | Ksink0 -
How can I use Google Analytics to detect users viewing my website on a TV?
I want to see in Google Analytics whether or not people are viewing my website on a TV, such as with a smart TV or other device connected to their TV. These are the only ways to do this that I have found so far: Operating system: Google TV, Nintendo Wii, Playstation 3, Xbox Browser: Nintendo Browser, Playstation 3 Are there other data points that I can reliably use to segment these users?
Reporting & Analytics | | RCF0 -
Are Vampires Haging Out On My Website
Buon pomeriggio from Wetherby multiple community flower bed winner 😉 On a site I'm working on web analytics is reporting traffic spiked in the early hours of the morning by a huge margin (this is so wrong). Yes its a multi national site but looking at the global distribution of visitors there really is no accounting for this. Here is a link to the data:
Reporting & Analytics | | Nightwing
http://i216.photobucket.com/albums/cc53/zymurgy_bucket/vampire-data_zpsc048684b.jpg Ive checked the analytics config & it is correctly set to GMT London so my question is:
"What could be causing traffic being incorectly logged in the early hours of the morning"? Grazie tanto,
David0 -
Can underscore blanks trigger home page bounce rates
Buongiorno from 8 degrees C Wetherby UK 🙂 On this site the scrolling banner has been hyperlinked with underscore blank causing new pages to open when a user clicks on a banner. My question is please... "If a user clicks on a banner will Google measure this as a home page bounce" ( I dont think it will i just want to be 100% sure) Click here for illustration:
Reporting & Analytics | | Nightwing
http://i216.photobucket.com/albums/cc53/zymurgy_bucket/home-page-sw-banner_zps0fda6318.jpg Grazie Tanto,
David0