Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
High Bounce Rate on traffic generating area of our site
Hi, Our eCommerce site currently includes a blog section known as Igloo which we have filled with unique and helpful content that is useful to a fair few people, not just customers of ours. It currently attracts a large number of visitors (more than the actual eCommerce side of the site in actual fact) organically who aren't currently customers of ours. Very few of these turn in to paying clients so it's not really a money spinner but it has worked quite well from a linkbait perspective / traffic generation perspective and undoubtedly a few of these people do end up making a purchase on the actual shopping end of our site. We're look at ways to encourage these people finding help on this free resource to take a look at our homepage and hopefully make an order but in the meantime I am worried that there may be a few downsides to us creating this content: Google may see us more as a help site than a shopping site. Since selling products is where we make our money this could ultimately be a bad thing. Our bounce rate is REALLY high (I'm talking around 94%) on the help site versus around 20% on the eCommerce site. I guess people land on the article they want, read it and then disappear. Would this bounce rate skew our entire site stats and ultimately result in decreased performance in the SERPS. I would appreciate your opinions and, in the event you do feel it may be hurting us overall perhaps some suggestions on how to mitigate the effects? Many thanks!
Reporting & Analytics | | ChrisHolgate0 -
How do I add subdomain tracking to an existing Google analytics account that was set up to track website only (without the subdomain option)
I know you can track subdomains by just selecting the proper code when you set up the analytics and then create filters for the data in analytics. But how do you add a subdomain for existing analytics website. Is there a way to go back and change to the option to include subdomains and then I assume just replace the tracking code with the new code that Google delivers for this?
Reporting & Analytics | | rhgraves650 -
Wordpress site with increase number of Crawl(400 response Code) errors in Others section of GWT
I have a wordpress site http://muslim-academy.com/I check in Google Webmasters tool today and I see the increase number of errors in Others area of Google webmaster Tool.The error code is 400http://muslim-academy.com/%D8%B3%D9%8A%D8%B1%D8%A9-%D8%AA%D8%A7%D8%B1%D9%8A%D8%AE%D9%8A%D8%A9-%D9%84%D9%84%D8%B1%D8%A6%D9%8A%D8%B3-%D8%AC%D9%85%D8%A7%D9%84-%D8%B9%D8%A8%D8%AF-%D8%A7%D9%84%D9%86%D8%A7%D8%B5%D8%B1-2/%D8%B3%D9%....%3Cbr%20/%3E________________%3Cbr%20/%3E___________%3Ca%20href=?lang=zhOne of the example link of this error.Can you guide me why the number of errors are increasing and how to fix the existing errors.
Reporting & Analytics | | csfarnsworth0 -
Automatic Checking indexation of websites
Hi Guys, do you know a tool that can check al list of websites (directorys) wich automatically checks if the website are indexed in Google. The list is very long and I would like to have a tool wich checks them all with only CnP them once. thankx a lot der.rabauke
Reporting & Analytics | | Lincus0 -
Subdomain and relative link paths cause crawl errors
I have a Wordpress blog on our subdomain and we use relative paths on our domain. It appears as though Google bot is crawling from the subdomain categories back to the domain relative paths. This of course results in hundreds of 404 pages. Any suggestions as to how to resolve this issue without changing the relative path structure of our domain? I can provide more information if need be. While I realize these issues are not that pressing, I'd obviously like to remove as many errors as possible. If anyone has encountered this problem, especially in Wordpress I'd really like to hear your solution or lack there of. Thank you in advance.
Reporting & Analytics | | BethA0 -
What should I expect after completely redoing a website?
Hello Forum, My team and I have completely redone an eCommerce website, which offers new SEO-friendly URLs. We have also taken care to optimize our product and category pages for specific keywords, so as to keep our regular traffic and increase the long-tail. In the old un-optimized website, many of its Google search results simply took users to the homepage, thus the homepage has a lot of external link juice. Our new site is optimized so that users are taken to the page most relevant to their query. For example (we are a yoga site), with the old site a search for "yoga bolsters" would take visitors to the homepage. Now the search results take the users to the yoga bolster page. These new pages don't have the amount of external links as the new site and I've noticed a decrease in our site's average rank in Google Webmaster Tools 1.Given our new site with its new URLs, should this increase be expected? (we have implemented 301 redirects from the old site to the new) 2. What steps would you take to ensure you keep your rankings with the new site? Are they any special tools you recommend? 3. What the best way to increase the ones that have dropped? Page optimization? Link building? Both?
Reporting & Analytics | | pano2 -
Pages crawled
Hi I've created a campaign for my own website and added 3 competitor sites. Under the campaign it says that 53 pages have been crawled but my site has less than 10 pages. Are the other pages from my competitor sites? Thanks James
Reporting & Analytics | | avecsys0 -
Bounce Rate
Howdy Mozzers Does anyone know if the 'average time on site' in Google analytics is calculated with bounce rate included? For example if you have a 50% bounce rate and your average time on site is 2 minutes the actual time would be 4 minutes as the 50% bounce rate time is classed as 0. I hope that is clear! Cheers
Reporting & Analytics | | CraigAddyman2