Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Increase in Direct Traffic plus Bounce Rate rise for all traffic sources
Hello, I work for an agency and we have seen a big rise in bounce rate for 4 of our clients which happened on the exact same day. This rise on bounce rate is across all traffic sources. We are also seeing a big increase in direct traffic, starting on the same day. Is it possible for bot traffic to affect the bounce rate of all other traffic sources? We have ruled out double reporting in GA but can explain how the bounce rate has increased for all traffic sources. How is this linked to the rise in direct traffic (in some cases as high as 500%)? Thanks
Reporting & Analytics | | jenallen0 -
Bounce Rate Extremely Low
Bounce rate for one of my client's website has drastically reduced. Its below 5%. Have ensured that GA code is placed only once. Any help or tips in identifying the root cause is much appreciated. Shall be happy to share the URL if required.
Reporting & Analytics | | seoninj0 -
Less than 7% bounce rate
I've just taken on a new client and on first look at their Google Analytics they are getting a bounce rate between 7% and 0%. Frankly I'm amazed if this is correct as between 1 April 2017 and 21 June 2017 they've driven 6403 session to the site from a range of referrers as well as AdWords (which has a 0% bounce rate), so traffic is pretty good. I've never seen such a low bounce rate which is making me suspicious, what can I do to check it's all recording properly? Any other ideas or have I found a near perfect site in terms of bounce? The url is http://knuma.co.uk/
Reporting & Analytics | | Marketing_Optimist1 -
How do you handle a "too many inbound links" notice on your crawl diagnostics report?
How do you handle a "too many inbound links" notice on your crawl diagnostics report? The url's identified come from our blog....is it reading all the blog post URL's and giving me a notice? Is this ok to leave all the links? Or would anyone recommend no follow tags on all of our blog posts?
Reporting & Analytics | | cschwartzel0 -
Open internal links in a new tab increase bonus rate?
Hello! This week I used a simple method to reduce my blog Google Analytics bounce rate. My blog all the posts are guides, in order to follow them, user need to download a zip file (same zip file). Otherwise they can't. Therefore I added a separate blog post to download all the necessary files. As a result of that I can reduce my bounce rate from 62-70% to 45-50% level. Now I'm thinking to open that zip file download page in a new tab. If I open my blog zip file download page, in a new tab. It will again increase my bounce rate? I reduced my bounce rate using that zip file download page. Thanks!
Reporting & Analytics | | Godad0 -
I have two campaigns that are only crawling one page, why is this?
I have a total of three campaigns running right now, and two of them are only crawling one page. I set the campaigns up the same, what is the problem?
Reporting & Analytics | | SiteVamp0 -
Why is a section of our website dropping in&out of Google SERPs?
In July 2011 we started a news section that has it's own 'subfolder' /news/ (http://www.chorder.com/news/new_gear/, http://www.chorder.com/news/gear_deals/ etc.) The whole news section is dropping in&out of Google SERP's since late October, as show in attached graph. All news texts are real deal, written by our own staff, linked from homepage. Any idea why this happens and how to prevent it? cmqky.png
Reporting & Analytics | | imventurer0 -
Pages crawled
Hi I've created a campaign for my own website and added 3 competitor sites. Under the campaign it says that 53 pages have been crawled but my site has less than 10 pages. Are the other pages from my competitor sites? Thanks James
Reporting & Analytics | | avecsys0