How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
410 /demandware.store/ pages
I have a client that just migrated to salesforce commerce cloud. They have a large amount of 410 pages showing up in site crawls. They are all of the /on/demandware.store/ variety. Is there any reason to redirect thousands of these pages? Any value? Or should they be allowed to drop off?
On-Page Optimization | | bdcseo1 -
No meta description on category page
Hi Moz is reporting no meta description on a wordpress category page like this one: http://www.dwliverpoolphotography.co.uk/category/uncategorized/ Can I add a meta description to a category? Best wishes. David.
On-Page Optimization | | WallerD0 -
How can i block the below URLs
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.? http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/ http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/ http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/ My robots.txt file is - User-agent: * Disallow: /wp-admin/
On-Page Optimization | | MasonBaker0 -
Duplicate Content aka 301 redirect from .com to .com/index.html
Moz reports are telling me that I have duplicate content on the home page because .com and .com/index.html are being seen as two pages. I have implemented 301 redirect using various codes I found online, but nothing seems to work. Currently I'm using this code. RewriteEngine On
On-Page Optimization | | omakad
RewriteBase /
RewriteCond %{HTTP_HOST} ^jacksonvilleacservice.com
RewriteRule ^index.html$ http://www.jacksonvilleacservice.com/ [L,R=301] Nothing is changing. What am I doing wrong? I have given it several weeks but report stays the same. Also according to webmasters tools they can't see this as duplicate content. What am I doing wrong?0 -
Nice looking ecommerce menus with featured product categories - bad for SEO due to duplicate content?
My ecommerce website has menus which contain 'featured product sub-categories'. These are shown alongside the other product sub-category links. Each 'featured product category' includes a link, an image (with link) and some text. All menu content is visible to search engines. These menus look nice and probably encourage CTR (not tested!) but are they bad for SEO?
On-Page Optimization | | Coraltoes771 -
Categories and URL Structure - When to add a new directory?
I've been wondering this for quite awhile so I figured I should just ask. Suppose my website has 5 categories and the url structure looks like: www.mysite.com/category1/ www.mysite.com/category2/ do I also want to create a landing page for the above categories at the same URL depth as the homepage of the site? www.mysite.com/category1.html OR what about: www.mysite.com/category1/index.html Which is a better way to do this? Also, if your site began as fairly small and your 5 categories were your only other pages other than index, about, and contact pages (meaning you really had no reason to create separate directories), then as time passes, you decide to add 3 subcategory pages that would fit into a page: www.mysite.com/category1.html would you add a folder with he same name as the html page, and then rename the html file as index.html and place it into the new folder?
On-Page Optimization | | SEO-Pump.com0 -
Ecommerce, Adding Content To Categories/Product Pages
In an eCommerce store, when is it appropriate to add quality category pages content and when is it more appropriate to add content to the actual product pages instead?
On-Page Optimization | | BobGW0 -
Title Tag, Internal Categories, Long tail
Hello, For internal categories - those that are not in the main navigation - we're optimizing for the long tail. How do you optimize the title tag for the long tail? For example if the internal category is about Men's shoes -> "Adidas Crosstrainers" and you already have a page in the main navigation with "Adidas Crosstrainers" in the title, what kind of title do you use for the long tail category? These categories are usually for the convenience of the user. I assume they should each have at least 500 words in them to do the best long tail searches. Thanks.
On-Page Optimization | | BobGW0