Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Drop in Bounce Rate in Google Analytics
Hi guys, I have recently seen a large drop in bounce rate (from GA) which seems unnatural for one of our clients website. Since the start of 2018, the bounce rate was consistently between 40-60%, and then saw a random spike, and now for the past two weeks, the bounce rate is below 10%. I was wondering if anyone had any ideas if this is a problem with GA, or the site itself. Site: https://www.zoomocarcredit.com/ Any comments/feedback is greatly appreciated. Thanks in advance! Jack. SnP0Hc4
Reporting & Analytics | | ChemistryMarketing0 -
Connect AMP to the same website GA property or not
Hi, Asking this on behalf of a client:"I know Google recommends setting up a separate GA property for AMP which they note in a few places:https://developers.google.com/analytics/devguides/collection/amp-analytics/https://support.google.com/analytics/answer/6343176?hl=en
Reporting & Analytics | | MediaCause
From what we can tell this is because the tracking for AMP will measure things differently than the current analytics.js library. I also think it's related to the GA cookie and how it's set differently for AMP pages.But won't this cause drop-off in user journey information? For example, if a user views an AMP blog article but then clicks to our site from a link within the article. I think we would then track this as two separate users and the pages/visit would be off. I found this article which explains a tech way to connect AMP to the same website GA property but it seems a little too much of a work-around:https://www.simoahava.com/analytics/google-analytics-client-id-amp-pages/ Do you think we should just do what Google recommends and set up AMP as a separate property in GA?" --> Does anyone know if there could be any issues with this workaround?Thanks,0 -
Bounce Rate % reduce after implementing enhance ecommerce
Hi Guys, My bounce rate is normal 47% to 50% but after implementing enhance ecommerce it is showing 28% dont know what is the reason?
Reporting & Analytics | | varo0 -
How can I see what Google sees when it crawls my page?
In other words, how can see the text and what not it sees from start to finish on each page. I know there was a site, but I can't remember it.
Reporting & Analytics | | tiffany11030 -
How do I add subdomain tracking to an existing Google analytics account that was set up to track website only (without the subdomain option)
I know you can track subdomains by just selecting the proper code when you set up the analytics and then create filters for the data in analytics. But how do you add a subdomain for existing analytics website. Is there a way to go back and change to the option to include subdomains and then I assume just replace the tracking code with the new code that Google delivers for this?
Reporting & Analytics | | rhgraves650 -
How can I use Google Analytics to detect users viewing my website on a TV?
I want to see in Google Analytics whether or not people are viewing my website on a TV, such as with a smart TV or other device connected to their TV. These are the only ways to do this that I have found so far: Operating system: Google TV, Nintendo Wii, Playstation 3, Xbox Browser: Nintendo Browser, Playstation 3 Are there other data points that I can reliably use to segment these users?
Reporting & Analytics | | RCF0 -
100% Bounce Rate in my Checkout??
I am currently building a report on our bounce rates for the past few years. When I look at older data, from 2010-11 I see varying bounce rates for the checkout, which is fine. When I move to new data, 2011-12 and 2012-13 I am seeing 100% bounce rates and exit rates for my checkout. We do use a 3rd party for our checkout, Shopatron and yes we do have it set up to track. Any thoughts on why I might be seeing this high rate for my bounce rate oh and this also goes for my exit percentage as well.
Reporting & Analytics | | K2_Sports0 -
Site crawler hasn't crawled my site in 6 days!
On 4.23 i requested a site crawl. My site only has about 550 pages. So how can we get faster crawls?
Reporting & Analytics | | joemas990