Why might my websites crawl rate....explode?

[Silkstream]

Hi Mozzers,

I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:

9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809

I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice.

[Silkstream]

Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!

I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.

I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.

I have no idea what caused it! Thanks for your help and advice though, all appreciated!

[BlueCorona]

Hi Silkstream!

One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.

I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?

Hope this helps!

[OlegKorneitchouk]

Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?

There appears to be an update rolling out which usually means increased crawler activity.