How do i block an entire category/directory with robots.txt?

[bricerhodes]

Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.

The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.

I'm not really sure how i'd type it into the robots.txt file, or where to place the file.

any help would be appreciated thanks

[bricerhodes]

Thanks for the detailed answer, i will give it a try!

[LynnPatchett]

Hi

This should do it, you place the robots.txt in the root directory of your site.

User-agent: * Disallow: /product-category/

You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt

As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.

Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).

It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.

Hope that helps!