Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
If website users don't accept GDPR cookie consent, does that prevent GA-GTM from tracking pageviews and any traffic from that user that would cause significant traffic decreases?
I've been doing a lot research on GDPR impact and implementation with GTM-GA for clients, but it's been 12 months since GDPR has gone live I haven't found anything on how GA traffic has been impacted if users don't accept cookie consent. However, I'm personally seeing GA accounts taking huge losses in traffic since implementing GDPR cookie solutions (because GTM/GA tags aren't firing until cookies are accepted). Is it common for websites to see significant decreases in traffic due to too many users not accepting cookie consent? Are there alternative solutions to avoid traffic loss like that and still maintain GDPR compliance? It seems to me that the industry underestimated how many people won't accept cookie consent. Most of the documentation and articles around GDPR's start (May 2018) didn't foresee or cover that aspect properly, everything seems to be technically focused with the assumption that if implemented properly most people would accept cookie consent, but I'm personally not seeing that trend and it's destroying GA data (lost traffic, minimal source attribution, inaccurate behavior data, etc). Thanks.
Reporting & Analytics | | Kickboard2 -
Moving an entire section of a website to a new URL
Hello there,
Reporting & Analytics | | CraigFairgrieve
I currently have a website that offers services to both domestic and business clients. Currently these are hosted on a single URL, then split out .com/business and .com/domestic We're going to be moving the .com/business structure and content to an entirely new URL which will be dedicated to only the business part of the business. This will also mean a change in the branding of the existing content as the new URL will come under the name of the parent company. My question is two fold: (1) What's the best way to go about this? (2) What would be the estimated effect on the traffic? Many thanks for your help in advance.0 -
How to track mobile actions on my website
Hi, i have a website that have, as many others, a mobile version. I want to know if there exist a way to track the mobile user experience, for example, how many times scroll down the page, where touch the screen, and actions that only have sense with mobile visitors. what do you think? Thanks, Juan Ignacio
Reporting & Analytics | | NachoRetta0 -
Webmaster tools crawl errors
Hi there, iv been tracking my webmaster tools crawl errors for a while now(6 months) and im noticing some pages that are far gone 404 are still poping out on the crawl errors. - that pages have no data for xml linking, and remote linking are from pages that are far gone 404 also. that pages have 404 error page + redirect to homepage, and google still notice them with old cache content. does someone have a clue why is this happening?
Reporting & Analytics | | Or.Shvartz0 -
How to track and verify 301 redirects from 22 old websites to a new one?
Me and my team have just finished a website which will replace 22 old ones (6 main domains and 16 sub-domains, 19 in wordpress and 3 in Joomla). This means more than 20,000 links that we're trying to redirect with the most granularity possible. But we're experiencing a lot of conflicts because of the quantity and variety of links and the interference of Wordpress. And I'm having trouble finding which links are working and which aren't. I have a URL list in a plain .txt file for every website, is there a tool that can check these links for me? I've tested many wordpress plugins, but I didn't find a good one. Thanks!
Reporting & Analytics | | bernardovailati0 -
If I am changing my domain for my website and want to keep using the same Google Analytics account to keep the data from the old domain. How should I proceed?
If I am changing my domain for my website and want to keep using the same Google Analytics account to keep the data from the old domain. How should I proceed? Do I have to start a new Google Analytics account for the new domain? If so how do I keep the old data? Or can I use the same GA account? Thank you.
Reporting & Analytics | | brianhughes1 -
Dramatic Increase in referrals from own website
The past few weeks I've been wracking my brain to figure out why on earth my branded searches could be dropping off at 30-40%. Well today, I realize I've had a dramatic increase in referrals from my own site (the non www verison). I'm talking 150 in March of last year to 5k in March of this year drastic. My 301 redirects haven't changed as far as I know -- I've had them set to redirect from the non www. to the www. for at least a year or two. I'm assuming visitors from search engines are somehow getting the non www version and the redirect is attributing the traffic to referrals instead of search. The drop in search traffic and the increase in referral traffic fall on the same day. Does that sound right/possible? If so, how do I fix this? My traffic stats for two clients are all screwy because of this. I want to make sure whatever solution I implement won't hurt my search traffic numbers any more 🙂 Has anyone else seen this happen recently? I could imagine an anomaly with one site but I find it odd that it could be two client sites. I still have some others to check. Thanks in advance! Leslie
Reporting & Analytics | | LeslieVS0 -
Site crawler hasn't crawled my site in 6 days!
On 4.23 i requested a site crawl. My site only has about 550 pages. So how can we get faster crawls?
Reporting & Analytics | | joemas990