Yahoo Slurp Bot 3.0 Going Crazy
-
On one of our sites, since the Summer, Yahoo Slurp bot has been crawling our pages at about 5 times a minute. We have put a crawl delay on it and it does not respect our robots.txt. Now the issue is it's triggering javascript (which bots shouldn't) triggering our adsense, ad server, analytics information, etc.
We've thought of banning the bot all together but get a good amount of Yahoo traffic. We've though about programmatic-ly not showing the javascript (ad + analytic) tags but are slightly afraid the Yahoo might consider this cloaking.
What are the best practices to deal with this bad bot.
-
I've searched the web but cannot find a specific support location. Any suggestions or links.
-
Bots do folow javascript links these days, maybe yahoo have jsut started to do so, maybe they are not doing so well at it.
I would contact Yahoo and try and get some answers.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Malicious bots
I was looking at some recommended keywords and felt sick to my stomach when I saw ilovevitaly.com search shell, resellerclub scam and a few more. | 2. | | 28(2.29%)ilovevitaly.com search shell | 0.00% | 0(0.00%) | 42.86% | 1.75 | 00:10:13 | 0.00% | 0(0.00%) | $0.00(0.00%) |
White Hat / Black Hat SEO | | BlueprintMarketing
| | 3. | resellerclub scam | I believe I have found the multiple IP addresses in which they're coming from and when I say many I mean I found 200 or so. There from different C blocks so they're very difficult to block easily without blocking legitimate traffic. I'm using a couple of different web application firewalls with the ability to block it pretty much anything. Does anyone have any device on doing this in a manner that might be more efficient than what I'm doing.I definitely do not want Google to think this is something that I did and penalize somebody this would be horrible. The site is going through Sucuri.net to be cleaned of any possible infection right now I do not know how this happened but zero day attacks are unfortunately a very real reality and unfortunately it could've been 1 million things. Thanks a million guys. I appreciate your help,
Tom0 -
Separate Servers for Humans vs. Bots with Same Content Considered Cloaking?
Hi, We are considering using separate servers for when a Bot vs. a Human lands on our site to prevent overloading our servers. Just wondering if this is considered cloaking if the content remains exactly the same to both the Bot & Human, but on different servers. And if this isn't considered cloaking, will this affect the way our site is crawled? Or hurt rankings? Thanks
White Hat / Black Hat SEO | | Desiree-CP0 -
Are Links from blogs with person using keyword anchor text a Penguin 2.0 issue?
Hello, I am continuing a complete clean up of a clients link profile and would like to know if Penguin is against links from blogs with the user including keywords as anchor text? So far I have been attempting to get them removed before I go for a disavow. An example would be the work clothing comment at the bottom of: http://www.fashionstyleyou.co.uk/beat-the-caffeine-rush.html/comment-page-1 I am also questioning if we should keep any link directories, so far I have been ruthless, but worry I will be losing a hell of a lot of links. For example I have kept the following: http://www.business-directory-uk.co.uk//clothing.htm Your comments are welcomed!
White Hat / Black Hat SEO | | MarzVentures0 -
Post-Penguin 2.0 Gust Blogging
I'm really just curious about everyone’s thoughts on post-Penguin 2.0 guest blogging. Is it still a viable option for link building? Is there anything you should proactively do to make it "safe"? What makes a guest blog post "advertorial" (or would it never be, if it is clearly marked as a guest post with a writer's bio)? Will moderate guest blogging on highly related, top ranked sites ever be a prime target for Google updates? I feel like guest blogging is still a viable way to build links, as long as it is on high quality and highly relevant sites that post content people actually read. Limit the number of links to 1-3 for every post, use generic or branded text as anchor text rather than your "top keyword" anchor text of old, and make the content interesting (educational or funny, not just for the sake of getting links) and completely unique to the site you are posting on. Just my 2 cents. Anyone else?
White Hat / Black Hat SEO | | jaredkipe0 -
Penalized by Penguin 2.0
I believe our site has been penalizes by Penguin 2.0. Our impressions in Google Webmaster are down and our traffic in Google Analytics also took a hit. Both of these occurences took place right when Penguin 2.0 was unleashed. What are the steps I need to take to regain my ranking? Is disallowing all the links I think maybe spammy the first thing to do?
White Hat / Black Hat SEO | | joebuilder0 -
How do I prepare my store for Penguin 2.0?
Looking for advice from someone who has been keeping up on the updates from Matt Cutts, other sources on what to work on for my web store to prevent getting hit hard by the upcoming penguin update.Practical advice on what to clean up on the site would be extremely useful. Watched a Matt Cutts video yesterday getting a preview...I'm very curious about the part saying that Google will show preference to those who are "Experts in their Fields." What makes you qualified for this? We are in the wicker furniture industry and have been a local brick and mortar store since 1982. We started our website about 1998, so I would consider that experience possibly part of the equation. On the other hand, I know everyone would love to say they are the experts in their niche, so it would be nice to know what needs to be done to achieve this. Thank you in advance for any help, Mark Grabowski Wicker Paradise xQmQeKU25zg
White Hat / Black Hat SEO | | wickerparadise0 -
How is this different than Go Daddy Spam?
Out of boredom, I googled up "SEO Company". #1 result is qualifiedimpressions.com. Taking a look at their link profile it seems they are utilizing anchor text on all their clients' websites. Moreover, it appears they have multiple sites for each of their phrases (which they cross link). Qualified Impressions - SEO Company WeBuildRankings.com - SEO Service VisibilitySquad - SEO Companies Some of their clients are rocking multiple anchor text links. How is this not any different than what Go Daddy did recently?
White Hat / Black Hat SEO | | ErikDster1 -
Beaten in SERP's by a site going 'all in' on 2 keywords in their anchor text profile.
I would like to get peoples thoughts on putting 80% of your anchor text links in just 2 keywords vs a nice spread of branded and longtail keywords.. like I am. recently fell off the first page for a key SERP.. and the site in P10 has gone nuts on just that two keyword's.. I know we have a good site onpage/ conversion / low bounce rate page views etc.. Pretty sure we get more traffic than them. Seems that this obvious bloated anchor text profiling has worked for them though.. What do you guys think/know?
White Hat / Black Hat SEO | | robertrRSwalters0