Why would I suddenly start seeing a spike in hits from particular bots (specifically rogerbot, google, bing, and yahoo)?

[eTundra]

We have seen consistent network traffic over the past month, then starting yesterday, huge spikes in hits (hits as in crawls to pages causing an increase in megabytes downloaded) started coming in from Rogerbot, Google, Bing, and Yahoo. A specific example from Rogerbot is as follows:

• rogerbot/1.1+(http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-crawler-104@moz.com)

• Useragent from the bot

• IP address: 54.226.73.52

• Domain / hostname: ec2-54-226-73-52.compute-1.amazonaws.com

• Physical location: United States flag United States, VA, Ashburn

We've have thought about doing a crawl-delay to prevent these bots from hitting us so hard, but that still doesn't help us answer why this even started in the first place.

Any clue on what may be going on here?

[CraigBradford]

Hi Kasy, did you get to the bottom of this?

[eTundra]

1. No changes yet; however, it's getting worse on our end, particularly from Yahoo, so we're about to update it to add a line for crawl-delay.
2. No known changes have been made with any of these.
3. No changes have been made to any of our canonical or noindex tags.
4. No, everything is the same.
5. The only one that we have consistently crawl the site is Moz.  I'm familiar with the other tools, but I haven't used them lately to crawl the site.

[CraigBradford]

Hi Kasy, sorry to check the obvious first...

1. Have there been any updates to your robots.txt file?
2. Have you updated sitemaps? In Robots.txt or Google webmaster tools?
3. Have you changed any meta information like canonical tags, noindex tags
4. Have you changed any internal links from no-follow to follow?
5. Have you got any tools regularly set up to crawl your site as Googlebot? Moz, Screaming Frog, DeepCrawl, Xenu etc.

[eTundra]

Nothing unusual in any of those areas.  GA is normal too.  The hits/bots did come all at the same time, but since it started, it's been consistent.

[LinkWheelOldSchool]

Anything strange in your link profile or in your social media profile. Did the hits/bots come all at the same time or are spread evenly within 24 hrs?