OK to block /js/ folder using robots.txt?

[AndreVanKets]

I know Matt Cutts suggestions we allow bots to crawl css and javascript folders (http://www.youtube.com/watch?v=PNEipHjsEPU)

But what if you have lots and lots of JS and you dont want to waste precious crawl resources?

Also, as we update and improve the javascript on our site, we iterate the version number ?v=1.1... 1.2... 1.3... etc.

And the legacy versions show up in Google Webmaster Tools as 404s. For example:

http://www.discoverafrica.com/js/global_functions.js?v=1.1
http://www.discoverafrica.com/js/jquery.cookie.js?v=1.1
http://www.discoverafrica.com/js/global.js?v=1.2
http://www.discoverafrica.com/js/jquery.validate.min.js?v=1.1
http://www.discoverafrica.com/js/json2.js?v=1.1

Wouldn't it just be easier to prevent Googlebot from crawling the js folder altogether?

Isn't that what robots.txt was made for?

Just to be clear - we are NOT doing any sneaky redirects or other dodgy javascript hacks.

We're just trying to power our content and UX elegantly with javascript.

What do you guys say:

Obey Matt? Or run the javascript gauntlet?

[katemorris]

Hey!

So, I listened to Matt's video. I see his point about wanting to crawl the JS files just in case something tricky is going on. Do understand that this is a risk you take. I don't see an issue blocking crawling of those files from a logical perspective, but if you or someone that takes over for you in the future does do something sneaky with JS and you are caught ... plus you have blacked access to the offending files ... it is going to take a lot more work to get back in good graces with them.

It's like a cop searching your car. You have every right to ban them from doing so, but if you have nothing to hide, why make trouble? Matt is right, banning crawling of these files is not going to save you much but if you think it's an issue, feel free. Just know that they might take it as a possible flag in the future.

Kate

[KeriMorgret]

Harald, it looks like the response you've quoted is from http://groups.google.com/a/googleproductforums.com/forum/#!category-topic/webmasters/crawling-indexing--ranking/9MGYEoROdkg, which is a question about a menu that has javascript. I think this poster has a slightly different question. I'll ask another associate to come on in and take a look.

[mediabase]

Hi Discover,I think that whenever we access the web pages , we have seen number of times that there is run time error & they asking for debug. This error message is helpful for the developers only but not for the users.

I think that you should please refer to the following link:

Blocking Javascript

The truth about non javascript

I hope that above content help to solve your query.