Yahoo Slurp Bot 3.0 Going Crazy
-
On one of our sites, since the Summer, Yahoo Slurp bot has been crawling our pages at about 5 times a minute. We have put a crawl delay on it and it does not respect our robots.txt. Now the issue is it's triggering javascript (which bots shouldn't) triggering our adsense, ad server, analytics information, etc.
We've thought of banning the bot all together but get a good amount of Yahoo traffic. We've though about programmatic-ly not showing the javascript (ad + analytic) tags but are slightly afraid the Yahoo might consider this cloaking.
What are the best practices to deal with this bad bot.
-
I've searched the web but cannot find a specific support location. Any suggestions or links.
-
Bots do folow javascript links these days, maybe yahoo have jsut started to do so, maybe they are not doing so well at it.
I would contact Yahoo and try and get some answers.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Key word Ranking are going down
Hi, my website's Keywords Ranking are going down, I am very new to SEO, I don't know what's the reason please help me out....my website is www.tourspackagedubai.com. Thanks
White Hat / Black Hat SEO | | tariqsalam0 -
Killed by penguin 3
So with the update to penguin 3.0 last week we notice that some clients have been significantly hit by the update. How do we rectify the situation for the poor links that are on the site. We have used open site explorer and Google webmaster to try and identify which are the bad links to try and remove. Now we can spot that some inbound links are from directories that may be perceived as low value/spam, but could not be sure what is affecting the ranking. The vast majority of these links are historical prior to inheriting this client recently and so do not have any logins to remove the links (if there are logins). These appear to be placed by teams outsourced in India. We would suspect that no site owner would spend the time removing links from the site any way. How do we recover from the penguin hit. Is it just a case of trying to identify ones that we suspect could be perceived as spam and ask for these to be disavowed by Google? Do we contact all the sites to ask them to be removed and/or do we just push ahead with more engaging white hat methods of social SEO? Are we likely to recover in the short term or be permanently hit. The site is for a small business with no more than 800 monthly hits so this fall from grace off very good front page positions is going to hit our client very hard even if the sins are from a previous business. Any thoughts and suggestions PLEASE HELP
White Hat / Black Hat SEO | | smartcow0 -
Do searchs bot understand SEF and non SEF url as the same ones ?
I've jsut realized that since almost for ever I use to code first my website using the non sef for internal linkings. It's very convenient as I'm sure that what ever will be the final url the link will always be good. ex: website.com/component1/id=1 Before releasing the website I use extensions to make the url user friendly according the choosen strategy. ex: website.com/component1/id=1 -> website.com/article1.html But I just wondered if google consider both urls as the same ones or if it consider just as a 301 redirection. What do you think is the best to do ?
White Hat / Black Hat SEO | | AymanH0 -
One of my outbound links website go hit by panda!
Hi mozzers, today I received a message from one of my blogger partners announcing me that he got hit by panda. 2 weeks ago I had him placing 2 anchors one in our main domain and a second one on our subdomain. I know panda focuses essentially on dups and I have paid attention to our webmaster tools to make sure we haven t got any messages Which we re good with. What do you guys suggest, will this affect us at some point or we re good? also in case that we re good will panda affect the blogger's authority therefore ours? if yes I should probably remove them, right? Thanks
White Hat / Black Hat SEO | | Ideas-Money-Art0 -
Can anyone explain these crazy SERPS?
do a UK based search for 'short term loans' on google. there are 7 sites on page 1 without any page or domain authority, several of them registered to a 'jeremy hughes', who I am guessing does not really exist. this is a very competitive term and they just shouldn't be making it onto page 1. im thinking this must be some clever 301 redirecting, as I cant see any backlinks to any of these sites in opensiteexplorer. any ideas how these sites are pulling this off?
White Hat / Black Hat SEO | | lethal0r0 -
My ranking dropped 3 pages on 18 november 2012
Hi There my site ranking dropped suddenly today for my main keywords such as security companies in london and security services in london from first page to 4th-5th page. these keywords were ranked on homepage http://www.armstrongsecurity.co.uk/ other keywords from some internal pages, such as this one http://www.armstrongsecurity.co.uk/security-services/event-security-london.html theygot hit slightly and went couple of listings down the road for event security london, event security companies london as well. same slight hitting happened on this page for main keywords http://www.armstrongsecurity.co.uk/bodyguard-for-hire-london.html can anyone help me, how to get the rankings back? my site authority is around 60 which is far better than most sites ranking higher than me now. these are some problems that i understand so far. keyword rich anchor text link profile for my main keywords over optimised pages let me know if anything you might find suspicious on my site that i can fix either on site or in my link profile. looking forward to your help. thanks gill
White Hat / Black Hat SEO | | spciuk0 -
Unnatural Link Notification - Third Go Round, specific questions
Hi all, I'm posting what is sure to be a common question, but I can't seem to find much information by searching Q&A over the last month so thought I'd throw this out there. There's a lot of 'what do I do??' questions about 'unnatural link notification', but most of them are from first timers. We're pretty far along in the process and it feels like we're going nowhere, so I was hoping to pick the brains of anyone else who's 'been there'. We have a client that we inherited with an unnatural link profile; they were warned shortly after we took them on (around March was the first warning). We compiled an apologetic letter, specifically identified a previous agency who >was< doing bad things, mentioned things would be different from now on, and provided a list of links we were working on to remove based on WMT and OSE and some other sources. This was submitted in early June. Traffic on the main keyword plummeted; ranking went from top 5 to about mid-page 4. We got hit with that same rash of Unnatural Link warnings on July 23 that everyone else did and after looking around I decided not to respond to those. We got a response to the reinclusion request submitted in June above, saying the site was still violating guidelines. This time I went all out, and provided a Google docs spreadsheet of the over 1,500 links we had removed, listed the other links that had no contact info (not even in WHOIS), listed the links we had emailed/contact formed but got no response, everything. So they responded to that recently, simply saying 'site still violates guidelines' with no other details, and I'm not sure what else I can do. The campaign above was quite an investment of resources and time, but I'm not sure how to most efficiently continue. I promised specific questions, so here they are: Are the link removal services (rmoov, removeem, linkdelete, et al) worth investigating? To remove the 1,500 links I mentioned above I had a full time (low paid) person working for a week. Does Google even reconsider after long engagements like this? Most of what I've read has said that inclusion gets cleared up on the first/second request, and we're at bat for the third now. Due to the lack of feedback I don't know if their opinion is "nope, you just missed some" or "you are so blackhat you shouldn't even bother asking anymore". One of the main link holders is this shady guy who runs literally thousands of directories the client appears in thanks to previous SEO agency, and wants $5 per link he removes. Should I mention this to Google, do they even care? Or is it solely our responsibility? Thanks in advance for any advice;
White Hat / Black Hat SEO | | icecarats0 -
Page Rank is 0
Hi. Can you please point me in the right direction concerning a site whose default page has a PR of 0? There does not appear to be any errors in the robots.txt file (that I can tell). When I ran a duplicate content check by searching the title tag and first sentance in quotes it did not return more than 2 sites. When I ran a site: it is reporting 287,000 results. Does this mean that they purchased links and have now been penalized? Or where should I go from here? Thank you for any feedback and assistance.
White Hat / Black Hat SEO | | JulB0