Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help Blocking Crawlers. Huge Spike in "Direct Visits" with 96% Bounce Rate & Low Pages/Visit.
Hello, I'm hoping one of you search geniuses can help me. We have a successful client who started seeing a HUGE spike in direct visits as reported by Google Analytics. This traffic now represents approximately 70% of all website traffic. These "direct visits" have a bounce rate of 96%+ and only 1-2 pages/visit. This is skewing our analytics in a big way and rendering them pretty much useless. I suspect this is some sort of crawler activity but we have no access to the server log files to verify this or identify the culprit. The client's site is on a GoDaddy Managed WordPress hosting account. The way I see it, there are a couple of possibilities.
Reporting & Analytics | | EricFish
1.) Our client's competitors are scraping the site on a regular basis to stay on top of site modifications, keyword emphasis, etc. It seems like whenever we make meaningful changes to the site, one of their competitors does a knock-off a few days later. Hmmm. 2.) Our client's competitors have this crawler hitting the site thousands of times a day to raise bounce rates and decrease the average time on site, which could like have an negative impact on SEO. Correct me if I'm wrong but I don't believe Google is going to reward sites with 90% bounce rates, 1-2 pages/visit and an 18 second average time on site. The bottom line is that we need to identify these bogus "direct visits" and find a way to block them. I've seen several WordPress plugins that claim to help with this but I certainly don't want to block valid crawlers, especially Google, from accessing the site. If someone out there could please weigh in on this and help us resolve the issue, I'd really appreciate it. Heck, I'll even name my third-born after you. Thanks for your help. Eric0 -
On Google Analytics, Pages that were 301 redirected are still being crawled. What's the issue here?
URL that we redirected are being crawled on Google Analytics. Since they dont exist, they have high bounce rates. What can the issue be?
Reporting & Analytics | | prestigeluxuryrentals.com0 -
Migrated website but Google Analytics still displays old URL's and none new?!
I migrated a website from a .aspx to a .php and hence had to 301 all the old urls to the new php ones. It's been months after and I'm not seeing any of the php pages showing results but I'm still getting results from the old .aspx pages. Has any one had any experience with this issue or knows what to do? Many thanks,
Reporting & Analytics | | CoGri0 -
Moz Crawl shows over 100 times more pages than my site has?
The latest crawl stats are attached. My site has just over 300 pages? Wondering what I have done wrong? RRv3fR0
Reporting & Analytics | | Billboard20120 -
*Dramatic* reduction in bounce rate, why?
Hi all I cannot pin this down to one of -new theme using Thesis 2 and Social Triggers, or -implementing Moz Does the Moz crawler linger on page? I'd love to know why this is happening 7iSnNfC
Reporting & Analytics | | TimMarsh0 -
Implications of Google discontinuing Website Optimizer
Hi Guys, As most of you probably know, Google is discontinuing Website Optimizer and introducing Experiments within Google Analytics. However, doesn't this mean that now, every site you want to run an experiment for has to be using Google Analytics? This is possibly one of the motivations for them making the change I guess? I also find it inconvenient that every 'experiment' now has to be based on improving a pre-defined goal in Google Analytics. This means that for a lot of situations we'll be creating goals just for the experiment and the clients actual goal conversions will appear quite inflated. I guess we'll just have to filter the new 'goals' out from the actual goals.
Reporting & Analytics | | David_ODonnell1 -
I am new to SEO. Would appreciate anyone share with me how can i further optimize my website to attract more visitors based on this scenario?
Hi all, Thanks and really appreciate for your time helping me out here, and i am very new to SEO. Just a brief introduction, our team have been developing a gaming review website (specifically targeting MMO game, facebook game, and browser-based game) since September 2011, and the website show significant improvement for the 1st month, and we got daily UV spiking up to 2k in November. The issue here is that the visitors start to become very stagnant, and even dropping now. Even though we try hard to publish more game reviews and there is NO IMPROVEMENT AT ALL for past 2 months in terms of number of visitors, and we have no idea how should we further proceed to attract more visitors...We have about 7 k site pages (however not all are with content). For the past 2 months it seems like there is nothing going on that proportionate to the effort we put in. Pretty discouraging, so we decided to sign up SEOMOZ to get SEO help from the community here. Some general questions we have: 1. Is it because of the the quality of the content we write ? (not enough keyword density, review too short, etc). 2. Is it because of the website designed not user friendly and not optimized? 3. Is it because of the general gaming trend, or other competitors? 4. Are we penalized by google by any chance? Are we spamming or anything wrong that deter our improvement? 5. Our main revenue source is on advertising, and currently we are using adsense. Would the ads placement affect anything? 6. Based on your knowledge and expertise, any advice would you give us? Would appreciate if you can take a kind look at the website to understand the scenario better: www.gameonline2.com, with 3 subdomain of: wiki.gameonline2; gameplay.gameonline2; cheats.gameonline2. For your reference, example our competitors are: www.onrpg.com; www.bbgsite.com; www.mmohut.com I would be happy to share with you any further information including google analytics if needed. I have report generated from SEOMOZ but i would appreciate the pros to help me take a look to give me a more complete ideas of how can i further improve to gain at least 10k visitors per day. Thanks again for your kind help and time. Andrew
Reporting & Analytics | | andrewsoh070 -
Pages crawled
Hi I've created a campaign for my own website and added 3 competitor sites. Under the campaign it says that 53 pages have been crawled but my site has less than 10 pages. Are the other pages from my competitor sites? Thanks James
Reporting & Analytics | | avecsys0