Block bad crawlers

[arielbortz]

Hi! how are you?

I've been working on some of my sites, and noticed that i'm getting lots of crawls by search engines that i'm not intereted in ranking well.

My question is the following: do you have a list of 'bad behaved' search engines that take lots of bandwidth and don´t send much/good traffic?

If so, do you know how to block them using robots.txt?

Thanks for the help!

Best wishes,

Ariel

[GeoffKenyon]

Hey Ariel,

Here's a couple lists of bots that some people are blocking - you should probably review your server data to see which bots are visiting you that you want to block:

• http://www.rhyolite.com/anti-spam/badbots.html
• http://searchenginewatch.com/robots.txt

In addition to the moz resource Chris referenced, here are a couple more pages that might be useful for you:

• http://stackoverflow.com/questions/10793906/how-to-allow-known-web-crawlers-and-block-spammers-and-harmful-robots-from-scann
• http://www.distilled.net/u/robots-txt/

Good luck!

[AlanMosley]

Chris gives a good answer, but is it really a problem, bandwidth is very cheap these days, in fact here in Australia most accounts are unlimited,

I Host with Microsoft Azure and bandwidth is very cheap.

[Chris.Menke]

Ariel, you could start with the list shown here and tailor it to fit your needs if you're having problems with others: http://www.webmasterworld.com/search_engine_spiders/4579553.htm.  There's info there on using robots.txt to block them and you should also read this for info on using robots.txt file: Robots.txt and Meta Robots - SEO Best Practices - Moz