Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My website has been declining in rankings!
Hello everyone, I have been doing the SEO on the following site: www.painters-decorators-london.co.uk for the past few years and the rankings are slowly declining since last year. (not the recent Google update). I can't seem to figure out what exactly is causing this to happen The page authority and domain authority is higher than my competitors. The on page SEO I believe is done appropriately but I'm not sure whats causing this decline. How can I find out exactly using the MOZ Pro account that I have. I don't think I'm totally aware of the important metrics that one can use on this site to determine the problem. Please advise!
Reporting & Analytics | | Ready2Paint0 -
Stripping referrer on website with a mix of both http and https
I know going from https to http (usually) strips referrers but I was wondering if the referrer is stripped when your website is a mix of both http and https? Say someone browses your site (on http), adds a product and then goes to your cart (https), then decides to go back to another page on your website which is http. Will this strip the referrer? Any help on this would be great, thanks!
Reporting & Analytics | | Fitto0 -
How to check bounce rate of mobile or ipad checkout steps? How to check Funnel for mobile or ipad?
Hello Guys, Is there any way to check bounce rate of mobile or ipad checkout steps till thankyou page? Also separate Funnel visualization for mobile or ipad? John
Reporting & Analytics | | varo0 -
What penalty might have hit here (screenshot attached)
I've had a spiky organic traffic profile for some time and wondered if anyone could suggest what penalty might have hit me. Something did in July but then when it recovered (not sure how) I ended up with double the traffic. Then it dipped, recovered a but and has slowly declined since. I've had no manual penalties so it must be algorithmic which means I can fix it. The problem is I don't know what the issues is! Any help appreciated! Ae00Lzl.png
Reporting & Analytics | | SamCUK0 -
How can I use Google Analytics to detect users viewing my website on a TV?
I want to see in Google Analytics whether or not people are viewing my website on a TV, such as with a smart TV or other device connected to their TV. These are the only ways to do this that I have found so far: Operating system: Google TV, Nintendo Wii, Playstation 3, Xbox Browser: Nintendo Browser, Playstation 3 Are there other data points that I can reliably use to segment these users?
Reporting & Analytics | | RCF0 -
Re-Launched Website: Developer Fogot to Remove noindex tags.
Our company's website has maintained decent rankings for the last 12 years we've been in business for our primary keywords. We recently had our website rebuilt from the ground up, and the developers left the noindex tags on all of our 400+ pages when we launched it. I didn't catch the error for 6 days. During which time, I used the Fetch feature in Google, submitting a site-wide fetch, as well as manual submissions for our top 100 URLs . In addition, every page that was indexed previously had a 301 set up for it, which was pointing to a destination with a noindex.
Reporting & Analytics | | yogitrout1
I caught the error today, and the developer removed the tags. Does anyone have any experience with a situation similar to this? In the SERPs, we are still ranking at this moment, and it's displaying our old URLs, and they are 301 redirecting just fine. But, what happens now? For 6 full days, we told Google not to index any of our pages, while also using the Fetch feature, contradicting ourselves.
Any words of wisdom or advice as to what I can do at this point to avoid potential fall out? Thanks0 -
Is it Possible to request G-WMT Resend emails pertaining to a certain website?
After being added as an administrator of a G-WMT profile, there are no messages from Google in the profile, however the first administrator has a number of messages. Is it possilble to request Google resends these emails so that I get them as well? I have looked around, and assume its not possible, but was wondering if someone had experienced this before and knew of a workaround. Greg
Reporting & Analytics | | AndreVanKets0 -
Implications of Google discontinuing Website Optimizer
Hi Guys, As most of you probably know, Google is discontinuing Website Optimizer and introducing Experiments within Google Analytics. However, doesn't this mean that now, every site you want to run an experiment for has to be using Google Analytics? This is possibly one of the motivations for them making the change I guess? I also find it inconvenient that every 'experiment' now has to be based on improving a pre-defined goal in Google Analytics. This means that for a lot of situations we'll be creating goals just for the experiment and the clients actual goal conversions will appear quite inflated. I guess we'll just have to filter the new 'goals' out from the actual goals.
Reporting & Analytics | | David_ODonnell1