Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's Causing My Extremely Low Bounce Rate
My client's site that is reporting an under 10% bounce rate for all sources. Direct is the highest at 8%. I'm no expert in GA but wondering if there is a problem with the analytics/tag manager code on the site. I'm especially concerned about the GTM body script being in an iframe which I read could be trouble. <!-- Google Tag Manager (noscript) -->
Reporting & Analytics | | bradsimonis
<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-MWGMNW6"
height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
<!-- End Google Tag Manager (noscript) --> You can see all the source code here:
view-source:https://nfinit.com/0 -
Bounce Rate Question - The percent calculated does not add up
Hello All, I'm attempting to see why organic search bounce rate has increased by 5% when compared to last year for a certain section of my website. I am using a custom segment to filter the specific pages I want to look at. Once the custom segment is set, I go to Acquisition - > Channels - > Organic. Then, I click the Landing Pages tab. Because we don't have keyword data anymore the only thing I can look at is the landing pages that contributed to the change in bounce. Finally, I set my date range and compare to the same date range as last year. Once I set the date range I am presented with a list of URLs and the percent change in bounce rate for each URL. This is where I get confused. If you look at the average bounce rate at the top of the column (example 1 attached) it does not add up with the data below it. If you export all of the data to excel, and then do an "Average" function in Excel, the data adds up to 17.29% instead of 35.04% for Sept. 2013. Why does this not add up? Isn't GA calculating the Average? Also, I always notice several URLs with only 1 session per URL. Several of these 1 session URLs have a 100% bounce rate. Since the bounce rate at the top of the column (example1) is a reflection of the average bounce rate, wouldn't these 1 session URLs significantly distort my data? I ultimately just want to see the pages that are contributing to the increased bounce rate when compared to last year. Having a hard time figuring this one out. Thank you all, Dave zMfAGls
Reporting & Analytics | | DaveGuyMan0 -
Conversion Rate Question: Should I Measure Visits or Unique Visits?
When you measure conversion rates, is the equation: conversion rate = visits/conversions or conversion rate = unique visits/conversions I ask because it can actually make a pretty big difference in the conversion rate. For example, if you visit my ecommerce website 100 times before buying something (and assuming you're my only visitor), then my conversion rate is 100% _if I'm determining conversion rates by unique visits/conversions. _However, it's only 1% _if I'm determining conversion rates by visits/conversions. _Wow! Now this is clearly an extreme example, but it should serve to illustrate the point that in more reasonable cases, the way the data is measured can have a potentially significant impact on the conversion rate. Is there an industry standard for this? Am I missing something really basic? Also, here's a little bit of context for the question: I run an ecommerce website powered by the Magento CMS and I'm trying to measure my conversion rate in Google Analytics for individual products. Google Analytics shows me my site wide conversion rate, but apparently I have to do some customization in order to measure conversion rates on the product level. That's fine, but I want to make sure I'm measuring my product conversions in a standard way. Thanks for any and all help! Adam
Reporting & Analytics | | Adam-Perlman0 -
I have two campaigns that are only crawling one page, why is this?
I have a total of three campaigns running right now, and two of them are only crawling one page. I set the campaigns up the same, what is the problem?
Reporting & Analytics | | SiteVamp0 -
Subdomain and relative link paths cause crawl errors
I have a Wordpress blog on our subdomain and we use relative paths on our domain. It appears as though Google bot is crawling from the subdomain categories back to the domain relative paths. This of course results in hundreds of 404 pages. Any suggestions as to how to resolve this issue without changing the relative path structure of our domain? I can provide more information if need be. While I realize these issues are not that pressing, I'd obviously like to remove as many errors as possible. If anyone has encountered this problem, especially in Wordpress I'd really like to hear your solution or lack there of. Thank you in advance.
Reporting & Analytics | | BethA0 -
Website not responding to web request
Hi, I'm attempting to create a campaign, but the website I want to analyse won't allow SEOmoz to crawl the site stating that the site does not allow 'web requests' - can anything be done about this? Thanks, Adam
Reporting & Analytics | | adamgthorndike0 -
My website traffic drop two times
Hi all, on our website www.watchalyzer.com I have unique content that we are writing especially for this online magazine. In last two months our traffic dropped two times. First time on October 20th and after 20 days traffic got back on November 10th. Second time traffic dropped on November 15th and it is still down Does somebody have idea what could be reason for this and how it can be fixed? thanks, Nikola
Reporting & Analytics | | GearyLSF0 -
Bounce Rates - How would you deal with this scenario?
Greetings! I actually don't have a definitive answer to this so wish to throw it out to the community for thoughts and feedback. I have a client who we shall call "Site 1", but they also have a job board, we shall call "Site 2". A product of their own success, they have a high bounce rate with visitors landing on Site 1, seeing a job they want to apply for and bouncing straight off to Site 2. The problem is that this is resulting in Google seeing some of these pages as having bounce rates of 80% to 100%, based on this formula: Bounce rate = total number of visits viewing only one page / total number of visits Now, I hate anything black hat or grey hat so wish to know how you would deal with this... If the results from Site 2 were displayed in a new framed page on Site 1, would this still be classed as a bounce? If when they click on a job on Site 1, they were taken to an intermediate page on Site 1 saying "Thank you, you are being redirected to your chosen job" for 5 seconds before being taken to Site 2, would this be classed as a bounce? Perhaps the job they wish to apply for 'pulled' from Site 2 and actually displayed in a new page on Site 1 would be a better way to go? I think that option 1 might work, sure that number 3 would but not so sure about number 2, but look forward to your comments and thoughts. Regards, Andy
Reporting & Analytics | | Andy.Drinkwater0