Yahoo Slurp Bot 3.0 Going Crazy
-
On one of our sites, since the Summer, Yahoo Slurp bot has been crawling our pages at about 5 times a minute. We have put a crawl delay on it and it does not respect our robots.txt. Now the issue is it's triggering javascript (which bots shouldn't) triggering our adsense, ad server, analytics information, etc.
We've thought of banning the bot all together but get a good amount of Yahoo traffic. We've though about programmatic-ly not showing the javascript (ad + analytic) tags but are slightly afraid the Yahoo might consider this cloaking.
What are the best practices to deal with this bad bot.
-
I've searched the web but cannot find a specific support location. Any suggestions or links.
-
Bots do folow javascript links these days, maybe yahoo have jsut started to do so, maybe they are not doing so well at it.
I would contact Yahoo and try and get some answers.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Top Keywords Ranking Going Down (continuously)
hi,
White Hat / Black Hat SEO | | seotoolshero06
I have an affiliate website where most of my articles are about product Reviews. And it was all going good before 15 february. After that Most of my keywords that were in top 3 places in SERP loses their position.
One of the keyword that was hurted badly after is "best coffee beans".
I have checked the search console and their is no warning issues or something suspicious. I have improved the onpage again, fixing every issue i got from audit tools, from speed to optimize images, titles to alt text, in short everything i could or that can be improved. But still coffee beans keyword is going down every passing day.
I am so worried and want you guys to please help me regarding this. Thanks2 -
Hiding ad code from bots
Hi. I have a client who is about to deploy ads on their site. To avoid bots clicking on those ads and skewing data, the company would like to prevent any bots from seeing any ads and, of course, that includes Googlebot. This seems like it could be cloaking and I'd rather not have a different version of the sites for bots. However, knowing that this will likely happen, I'm wondering how big of a problem it could be if they do this. This change isn't done to manipulate Googlebot's understanding of the page (ads don't affect rankings, etc.) and it will only be a very minimal impact on the page overall. So, if they go down this road and hide ads from bots, I'm trying to determine how big of a risk this could be. I found some old articles discussing this with some suggesting it was a problem and others saying it might be okay in some cases (links below). But I couldn't find any recent articles about this. Wondering if anybody has seen anything new or has a new perspective to share on this issue? Is it a problem if all bots (including Googlebot) are unable to see ads? https://moz.com/blog/white-hat-cloaking-it-exists-its-permitted-its-useful
White Hat / Black Hat SEO | | Matthew_Edgar
https://www.webmasterworld.com/google/4535445.htm
https://www.youtube.com/watch?v=wBO-1ETf_dY0 -
Apparent Bot Queries and Impressions in Webmaster Tools
I've been noticing some strange stats in Google Webmaster Tools for my forum, which has been getting spam queries with impressions and no clicks. See the queries in the attached images. This might be a motive for the spammers or scrapers. I set the date range to just 22 Aug - 22 Nov and I see very obviously the spike is due to impressions. Questions: What should/can I do? Is Google doing something about this? How to avoid this? o6gKB
White Hat / Black Hat SEO | | SameerBhatia0 -
Bot or Virus Creating Bad Links?
Hey Everyone, We are getting ready to engage a client for some potential marketing/SEO so in preparing for this have ran the site through OpenSiteExplorer. The site is relatively new and there are only two links under the inbound links section. They are relevant and add value, no issues there. Here is where it get strange. When I look under the 'Just Discovered' section there are many (hundreds) new links going back about a month. Virtually all of them have the anchor text 'Louis Vuitton outlet'. Now the client swears he has not engaged anyone for black hat SEO, so wondering who could possibly be creating these links. They do sell some Louis Vuitton items on the site, so I'm wondering if it is possible that some spam bot has picked up the site and began to spam the web with links to the clients site. So far today, 50 or so new links have been created with said anchor text and the clients root URL all on very poor quality, some foreign blog sites. Would like to find out why this is happening and put a stop to it for obvious reasons. Has anyone experienced something similar? Could this be a bot? Or maybe someone with an axe to grind against the client? Anyone could be doing this on their own, but just seems strange for it to be happening to a new site that does not even rank highly at the moment. Any advice or info is greatly appreciated, thanks in advance.
White Hat / Black Hat SEO | | Whebb0 -
Are Links from blogs with person using keyword anchor text a Penguin 2.0 issue?
Hello, I am continuing a complete clean up of a clients link profile and would like to know if Penguin is against links from blogs with the user including keywords as anchor text? So far I have been attempting to get them removed before I go for a disavow. An example would be the work clothing comment at the bottom of: http://www.fashionstyleyou.co.uk/beat-the-caffeine-rush.html/comment-page-1 I am also questioning if we should keep any link directories, so far I have been ruthless, but worry I will be losing a hell of a lot of links. For example I have kept the following: http://www.business-directory-uk.co.uk//clothing.htm Your comments are welcomed!
White Hat / Black Hat SEO | | MarzVentures0 -
What has been updated on part of Google Penguin 2.0?
I am looking for more details of Google Penguin 2.0 update. Is any information from SEO experts?
White Hat / Black Hat SEO | | gbavadiya1 -
Old SPAM tactic still works and gets TOP 3 in SERP?
Hi Mozers, Below you can see some examples of spam ( hidden text and sneaky redirects) which are in SERP for our branded keywords during last 3 months. Some of them occupy very high position in SERP (top 3/top5). https://www.google.com/search?num=100&newwindow=1&safe=off&biw=1883&bih=1028&q=%22your+mac+-%22%2B%22cleanmymac%22 I sent spam reports and I’m going to continue doing so. (~500 spam reports from personal and work google account) I contacting directly with some of the hacked sites (web-masters) and tried to help them to fix this issue, but it takes a lot of my time. But 3 months!? Can you give me any advice, what doing next? Thank you!
White Hat / Black Hat SEO | | MacPaw0 -
Page Rank is 0
Hi. Can you please point me in the right direction concerning a site whose default page has a PR of 0? There does not appear to be any errors in the robots.txt file (that I can tell). When I ran a duplicate content check by searching the title tag and first sentance in quotes it did not return more than 2 sites. When I ran a site: it is reporting 287,000 results. Does this mean that they purchased links and have now been penalized? Or where should I go from here? Thank you for any feedback and assistance.
White Hat / Black Hat SEO | | JulB0