Why would I suddenly start seeing a spike in hits from particular bots (specifically rogerbot, google, bing, and yahoo)?
-
We have seen consistent network traffic over the past month, then starting yesterday, huge spikes in hits (hits as in crawls to pages causing an increase in megabytes downloaded) started coming in from Rogerbot, Google, Bing, and Yahoo. A specific example from Rogerbot is as follows:
-
rogerbot/1.1+(http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-crawler-104@moz.com)
-
Useragent from the bot
-
IP address: 54.226.73.52
-
Domain / hostname: ec2-54-226-73-52.compute-1.amazonaws.com
-
Physical location: United States flag United States, VA, Ashburn
We've have thought about doing a crawl-delay to prevent these bots from hitting us so hard, but that still doesn't help us answer why this even started in the first place.
Any clue on what may be going on here?
-
-
Hi Kasy, did you get to the bottom of this?
-
- No changes yet; however, it's getting worse on our end, particularly from Yahoo, so we're about to update it to add a line for crawl-delay.
- No known changes have been made with any of these.
- No changes have been made to any of our canonical or noindex tags.
- No, everything is the same.
- The only one that we have consistently crawl the site is Moz. I'm familiar with the other tools, but I haven't used them lately to crawl the site.
-
Hi Kasy, sorry to check the obvious first...
- Have there been any updates to your robots.txt file?
- Have you updated sitemaps? In Robots.txt or Google webmaster tools?
- Have you changed any meta information like canonical tags, noindex tags
- Have you changed any internal links from no-follow to follow?
- Have you got any tools regularly set up to crawl your site as Googlebot? Moz, Screaming Frog, DeepCrawl, Xenu etc.
-
Nothing unusual in any of those areas. GA is normal too. The hits/bots did come all at the same time, but since it started, it's been consistent.
-
Anything strange in your link profile or in your social media profile. Did the hits/bots come all at the same time or are spread evenly within 24 hrs?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why has my search traffic suddenly tanked?
On 6 June, Google search traffic to my Wordpress travel blog http://www.travelnasia.com tanked completely. There are no warnings or indicators in Webmaster Tools that suggest why this happened. Traffic from search has remained at zero since 6 June and shows no sign of recovering. Two things happened on or around 6 June. (1) I dropped my premium theme which was proving to be not mobile friendly and replaced it with the ColorMag theme which is responsive. (2) I relocated off my previous hosting service which was showing long server lag times to a faster host. Both of these should have improved my search performance, not tanked it. There were some problems with the relocation to the new web host which resulted in a lot of "out of memory" errors on the website for 3-4 days. The allowed memory was simply not enough for the complexity of the site and the volume of traffic. After a few days of trying to resolve these problems, I moved the site to another web host which allows more PHP memory and the site now appears reliably accessible for both desktop and mobile. But my search traffic has not recovered. I am wondering if in all of this I've done something that Google considers to be a cardinal sin and I can't see it. The clues I'm seeing include: Moz Pro was unable to crawl my site last Friday. It seems like every URL it tried to crawl was of the form http://www.travelnasia.com/wp-login.php?action=jetpack-sso&redirect_to=http://www.travelnasia.com/blog/bangkok-skytrain-bts-mrt-lines which resulted in a 500 status error. I don't know why this happened but I have disabled the Jetpack login function completely, just in case it's the problem. GWT tells me that some of my resource files are not accessible by GoogleBot due to my robots.txt file denying access to /wp-content/plugins/. I have removed this restriction after reading the latest advice from Yoast but I still can't get GWT to fetch and render my posts without some resource errors. On 6 June I see in Structured Data of GWT that "items" went from 319 to 1478 and "items with errors" went from 5 to 214. There seems to be a problem with both hatom and hcard microformats but when I look at the source code they seem to be OK. What I can see in GWT is that each hcard has a node called "n [n]" which is empty and Google is generating a warning about this. I see that this is because the author vcard URL class now says "url fn n" but I don't see why it says this or how to fix it. I also don't see that this would cause my search traffic to tank completely. I wonder if anyone can see something I'm missing on the site. Why would Google completely deny search traffic to my site all of a sudden without notifying any kind of penalty? Note that I have NOT changed the content of the site in any significant way. And even if I did, it's unlikely to result in a complete denial of traffic without some kind of warning.
Technical SEO | | Gavin.Atkinson1 -
No google traffic for this site? Help?
Hi We have not done this web site http://climateacs.co.uk but have now picked it up and its getting no traffic what so ever from google do you think its been blacklisted? I have added it to my webmaster tools and there are no manual actions on it and most of the backlinks on google webmaster tools are from yell.com. However when I run it on opensiteexplorer I am seeing some chinese type links?? It is not really showing many search queries at all when you view them in webmaster tools under United Kingdom. I was going to start citation building for the address to help support the google places entry but just wanted to see what other peoples opinion was really on this site? Thanks Tracy
Technical SEO | | dashesndots0 -
Anyone See This Before? Google Following Links that are Not Hyperlinks
Today I was going through my Google Webmaster URL Errors (404s) info. I came across two links in my URL Errors report that are NOT actually hyperlinks on the source page. Both of these links are from two different forum-type websites. In both cases, the post references a URL on my website (incorrectly, hence the 404 error) in the text of the post but did NOT actually link to my site. I looked at the source code...no href. Both forum posts simply had a tag or tag around the incorrect URL text referencing my site. I have never seen this before or heard that Google will follow a URL that is not actually a hyperlink. Anyone else?
Technical SEO | | cajohnson0 -
Pages removed from Google index?
Hi All, I had around 2,300 pages in the google index until a week ago. The index removed a load and left me with 152 submitted, 152 indexed? I have just re-submitted my sitemap and will wait to see what happens. Any idea why it has done this? I have seen a drop in my rankings since. Thanks
Technical SEO | | TomLondon0 -
Google Analytics
I usually have Google analytics "real-time" running on one of my monitors, occasionally I glance at the screen to see that are TOP KEYWORDS people are using, lately there have been a lot of long-tail keywords. If I try to copy and paste the queries into google, i can never seem to find us organically for the long-tail searches? Is the real-time feature accurate? Thank you!
Technical SEO | | TP_Marketing0 -
Influencing Google Instant Preview
Hello there! I have been looking at how our articles are shown in Google Instant Previews. While the description in the SERPs picks up the first paragraph from the article from the main table/div body, the Instant Preview highlights a text ad in the top right of the side column. According to Google https://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews Q: How can I influence the text highlighted in the preview image?
Technical SEO | | CleverPhD
A: The highlighted text is automatically chosen based on the user's search query. Only text that is visible on the page can be selected for highlighting. Google is highlighting the same block of text from an ad space over and over again. "Download our Free Guide ...." vs the first paragraph in the article. Any ideas folks?0 -
Websites not being included on google from mobiles?
Hi, Just had a call from a guy saying that google have made a statement saying that it will be stopping people finding websites from mobule devices if they dont have a mobile domain name. Doesn anyone know anything about any Google statements or is this just rubbish?
Technical SEO | | Ant710 -
Google & Separators
This is not a question but something to share. If you click on all of these links and compare the results you will see why _ is not a good thing to have in your URLs. http://www.google.com/search?q=blue http://www.google.com/search?q=b.l.u.e http://www.google.com/search?q=b-l-u-e http://www.google.com/search?q=b_l_u_e http://www.google.com/search?q=b%20l%20u%20e If you have any other examples of working separators please comment.
Technical SEO | | Dan-Petrovic3