Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why few pages have more than 100% bounce rate?
Hello All, For my ecommerce site approx more than 30k products I have. In Google Analytic approx daily for few products approx 10-15 products bounce rate show 300%, 200%, 150%, 140%, 125% how? and what is the solution? Product page daily change. Thanks!
Reporting & Analytics | | Johny123450 -
When my new website went live I saw a huge spike in traffic for 3 weeks but then it halved again. Why would this happen?
My new website went live on 13th April and the organic traffic levels increased by 110%, staying at that level for 3 weeks. Then exactly 3 weeks later the organic traffic halved again, going back to the same level as the old site. Has anyone seen this happen before or worked out why?! Thanks Viki P2R76O9
Reporting & Analytics | | TWB0 -
How does Google sort multiple websites in one GWT account?
Today I noticed one of our sub-domains listed in our Google Webmaster Tools Account moved from the 6th position to the 2nd position. Is there a reason for this (perhaps urls/sites listed at the top require the most attention)?
Reporting & Analytics | | Prospector-Plastics0 -
Why would a website rank lower than weaker site?
Hi, Today I noticed that my website is ranking one place lower than a competitor in Google UK ,despite my site having a stronger domain authority and page authority. Is there a plausible reason for this, i'm slightly confused? Thanks,
Reporting & Analytics | | Benjamin3790 -
Best way to generate analytics reports for listing style website
I'm working on a website that includes dedicated pages for ~40 local businesses, and I need to be able to generate and export some basic reports I can send to each business. The data I need for each report is split between general sitewide data: total number of visitors to the site for month. number of visitors to each main category page what country they are from (main countries) - top 5 traffic source / keywords avg time on site As well as specific data for each individual page: how many people viewed specific total pages Time spent on individual page. Would this be possible with custom reports in analytics? I can see the number of different reports being difficult to maintain, especially as site grows. Anyone had expereince on a similar site of ideas on the best way to do this? Thanks
Reporting & Analytics | | zeald0 -
Is there a tool to automatically gather website SEO data?
I am looking for a tool that will crawl a website and create a spreadsheet listing out all key data such as title, meta description, etc. Anyone know of an available tool to do that?
Reporting & Analytics | | jfeitlinger0 -
What should I expect after completely redoing a website?
Hello Forum, My team and I have completely redone an eCommerce website, which offers new SEO-friendly URLs. We have also taken care to optimize our product and category pages for specific keywords, so as to keep our regular traffic and increase the long-tail. In the old un-optimized website, many of its Google search results simply took users to the homepage, thus the homepage has a lot of external link juice. Our new site is optimized so that users are taken to the page most relevant to their query. For example (we are a yoga site), with the old site a search for "yoga bolsters" would take visitors to the homepage. Now the search results take the users to the yoga bolster page. These new pages don't have the amount of external links as the new site and I've noticed a decrease in our site's average rank in Google Webmaster Tools 1.Given our new site with its new URLs, should this increase be expected? (we have implemented 301 redirects from the old site to the new) 2. What steps would you take to ensure you keep your rankings with the new site? Are they any special tools you recommend? 3. What the best way to increase the ones that have dropped? Page optimization? Link building? Both?
Reporting & Analytics | | pano2 -
Goal Conversion Rate increase AND Abandonment Increase
I wonder if someone can help me understand some data I'm seeing in Google Analytics. SetUp Goal with 3 step Funnel Date Comparison is 1st-8th September & 1st-8th June The Data (pics attached) Looking at the Goal Conversion Rate I can see: September - 11.58% vs June - 10.42% an increase which Google shows as (+11.18%) For same time period Abandonment Rate is showing September - 71.3% vs June - 61.5% - an increase which Google shows as (+16.5%) The Questions How can I bee seeing both an increase in conversion rate AND and increase in Abandonment rate on the funnel? And why doesnt "(Conversion Rate + Abandonment Rate) = 100%" ? Many thanks for any help! yqwFP.png cj5dn.png
Reporting & Analytics | | TobiasM0