Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should we rename and update a page or create a new page entirely?
Hi Moz Peoples! We have a small site with a simple site navigation, with only a few links on the nav bar. We have been doing some work to create a new page, which will eventually replace one of the links on the nav bar. The question we are having is, is it better to rename the existing page and replace its content and then wait for the great indexer to do its thing, or perm delete the page and replace it with the new page and content? Or is this a case where it really makes no difference as long as the redirects are set up correctly?
On-Page Optimization | | Parker8180 -
Recommended Schema for a Collection/Category page?
Hi There! Taking on a small project up updating and adding in Schema to a clients site; a previous developer half put in data vocabulary. In my planning I was wondering if their would be a best schema type for category page of products - or a collection of products? Any ideas and experience? Thanks!
On-Page Optimization | | paul-bold0 -
How to Handle duplicate pages/titles in Wordpress
The wordpress blog causes problems with page titles. If you go to the second page of blog posts it there's a different URL but with the same page title. for example: page 1: site/blog page 2: site/blog/page/2 Each page gets flagged for duplicate page titles. Thanks in advance for your thoughts,
On-Page Optimization | | heymarshall1 -
No meta description on category page
Hi Moz is reporting no meta description on a wordpress category page like this one: http://www.dwliverpoolphotography.co.uk/category/uncategorized/ Can I add a meta description to a category? Best wishes. David.
On-Page Optimization | | WallerD0 -
Do Parent Categories Hurt SEO?
I have parent categories and subcategories. Will it be harder for the subcategories to rank well because they have a parent category? The URL is longer, for one. I am just wondering if I should not have parent categories. I have one category page doing really well and I am trying to boost the others (most of which are subcategories) and this is a concern for me. Thanks! Edit: I also have a category that has 2 parent categories. I want it automatically in those 2 categories and one of its own. By itself it is very important keyword. Is this ok or should I have it be a parent category?
On-Page Optimization | | 2bloggers0 -
Ecommerce On-Site SEO: Keywords in Category Descriptions
Hello, I'm doing on-site SEO for a client's ecommerce site. Are 160 words enough for a category description? I'm using the keywords once at the top of the description, and once at the bottom of the description, with the ones at the bottom reworded so that they are the keywords with a different word order. I used to put the keywords in 3 times but it just feels like stuffing. Is twice, worded differently the second time, enough for a category description? Thanks.
On-Page Optimization | | BobGW0 -
Does Google index dynamically generated content/headers, etc.?
To avoid dupe content, we are moving away from a model where we have 30,000 pages, each with a separate URL that looks like /prices/<product-name>/<city><state>, often with dupe content because the product overlaps from city to city, and it's hard to keep 30,000 pages unique, where sometimes the only distinction is the price & the city/state.</state></city></product-name> We are moving to a model with around 300 unique pages, where some of the info that used to be in the url will move to the page itself (headers, etc.) to cut down on dupe content on those unique 300 pages. My question is this. If we have 300 unique-content pages with unique URL's, and we then put some dynamic info (year, city, state) into the page itself, will Google index this dynamic content? The question behind this one is, how do we continue to rank for searches for that product in the city-state being searched without having that info in the URL? Any best practices we should know about?
On-Page Optimization | | editabletext0 -
Blog.mysite.com or mysite.com/blog?
Hi, I'm just curious what the majority think of what's the best way to start a blog on your website for SEO benefits. Is it better to have it under a sub domain or a directory? Or does it even matter?
On-Page Optimization | | truckguy770