Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website Essentially Delisted and Top SERPS Removed
So what a morning.... Did a search for our website and looked to see how we were doing with a couple of search terms and it turns out our website is GONE. I mean not a slap down or fluctuations in positioning but we are literal gone. I see that the contact page is pulls up on page 8 but effectively we dont exist. This is so odd and strange that i feel that this is a dream. The fact that we dont pull up on any search terms anymore is of a concern that this is something more than a penalty. What on Earth could have caused this? This is fluke? Can i call Google? Seems that index page has been removed from the search results Website www.catdi.com Catdi Printing Search Terms Houston Printing Houston Direct Mail EDDM Printing EDDM Help! C
Reporting & Analytics | | CatdiPrinting0 -
Why only a few pages of my website are being indexed by google
Our website www.navisyachts.com has in its sitemap over 3000 pages of information, and this is all unique content written by our team. Now Google Webmaster central shows only 100 urls indexed from 3500 submitted. Can you help me understand why and how I can fix this issue? The website has 4 years old, is a Joomla 3.3 up to date. It has part of the content in the Joomla core content systems and part in K2. Thank you. Pablo
Reporting & Analytics | | FWC_SEO0 -
Google Crawl Stats
Hi all Wondering if anyone could help me out here. I am seeing massive variations in WMT of google crawl stats on a site I run. Just wondering if this is normal (see attached image). The site is an eccommerce site and gets a handful of new products added every couple of weeks. The total no of products is about 220k so this is only a very small %. I notice in WMT I have an amber warning under Server connectivity. About 10 days back I had warnings under DNS, Server and Robots. This was due to bad server performance. I have since moved to a new server and the other two warnings have gone back to green. I expect the Server connectivity one to update any day now. Ive included the graph for this incase it is relevant here. Many thanks for assistance. Carl crawlstats.png connect.png
Reporting & Analytics | | daedriccarl0 -
Bounce Rate Question - The percent calculated does not add up
Hello All, I'm attempting to see why organic search bounce rate has increased by 5% when compared to last year for a certain section of my website. I am using a custom segment to filter the specific pages I want to look at. Once the custom segment is set, I go to Acquisition - > Channels - > Organic. Then, I click the Landing Pages tab. Because we don't have keyword data anymore the only thing I can look at is the landing pages that contributed to the change in bounce. Finally, I set my date range and compare to the same date range as last year. Once I set the date range I am presented with a list of URLs and the percent change in bounce rate for each URL. This is where I get confused. If you look at the average bounce rate at the top of the column (example 1 attached) it does not add up with the data below it. If you export all of the data to excel, and then do an "Average" function in Excel, the data adds up to 17.29% instead of 35.04% for Sept. 2013. Why does this not add up? Isn't GA calculating the Average? Also, I always notice several URLs with only 1 session per URL. Several of these 1 session URLs have a 100% bounce rate. Since the bounce rate at the top of the column (example1) is a reflection of the average bounce rate, wouldn't these 1 session URLs significantly distort my data? I ultimately just want to see the pages that are contributing to the increased bounce rate when compared to last year. Having a hard time figuring this one out. Thank you all, Dave zMfAGls
Reporting & Analytics | | DaveGuyMan0 -
Improving Search Click through Rate
We are having a problem on our website with click through rates. We are getting between 100-150k impressions through search but we are only getting between 500-1000 clicks to the site. What strategies have you used in the past to help improve your click through rates? Thanks!
Reporting & Analytics | | pdangermond2 -
Is it Possible to request G-WMT Resend emails pertaining to a certain website?
After being added as an administrator of a G-WMT profile, there are no messages from Google in the profile, however the first administrator has a number of messages. Is it possilble to request Google resends these emails so that I get them as well? I have looked around, and assume its not possible, but was wondering if someone had experienced this before and knew of a workaround. Greg
Reporting & Analytics | | AndreVanKets0 -
Dramatic Increase in referrals from own website
The past few weeks I've been wracking my brain to figure out why on earth my branded searches could be dropping off at 30-40%. Well today, I realize I've had a dramatic increase in referrals from my own site (the non www verison). I'm talking 150 in March of last year to 5k in March of this year drastic. My 301 redirects haven't changed as far as I know -- I've had them set to redirect from the non www. to the www. for at least a year or two. I'm assuming visitors from search engines are somehow getting the non www version and the redirect is attributing the traffic to referrals instead of search. The drop in search traffic and the increase in referral traffic fall on the same day. Does that sound right/possible? If so, how do I fix this? My traffic stats for two clients are all screwy because of this. I want to make sure whatever solution I implement won't hurt my search traffic numbers any more 🙂 Has anyone else seen this happen recently? I could imagine an anomaly with one site but I find it odd that it could be two client sites. I still have some others to check. Thanks in advance! Leslie
Reporting & Analytics | | LeslieVS0 -
How do you add Facebook Insights to Multiple Websites?
If I have 4 top-level domain websites all for the same corporate company but with different focuses (ie/ engineering, technology etc.) and I want to add Facebook insights, should I add the same meta tag to all websites or should I create separate meta tags? If I create seperate ones, the issue is that we have one main facebook page that has multiple admins that I would like to give access to but it looks like you can only associate one website to one facebook page. So from what I can tell, if I were to create separate ones, I would have to setup fake facebook pages or fake profiles in order to get different facebook insights meta tags? I'm hoping there is a better way... or maybe I should just put the same meta tag code on all 4 websites? Thanks for your help!
Reporting & Analytics | | randstadsocial0
