Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best schema option for condos / condominiums?
Hey guys, I'm doing a review on some schema on some of our sites. Most of them are generic using LocalBusiness. There are a few more specific schemas I could use, but not sure what would be the most relevant. Wondering if any of you have a suggestion or ideas? https://schema.org/Residence https://schema.org/LodgingBusiness https://schema.org/ApartmentComplex or I could just stick with LocalBusiness. I'm leaning towards LodgingBusiness or ApartmentComplex.... but when I think of LodgingBusiness I think of something temporary / vacation type deal like hotels. Apartments... kind of self explanatory, a condominium isn't exactly an apartment but perhaps it is more comparable to an apartment than a hotel, motel or inn. What are you thoughts on this? Also, which "format" is better to use RDFa, microdata, or JSON-LD. Does it matter?
On-Page Optimization | | donnieath0 -
Random /feed 404 error from a wordpress site
My Moz Analytics report shows a 404 error on a page which I think should not exist at all. The URL is http://henryplumbingco.com/portfolio-item/butler-elementary/feed/. When I checked webmaster tools, it looks like there are a number of random /feed urls throwing 404 errors. I am using WordPress and the Enfold theme. Anyone know how to get rid of these errors? Thanks,
On-Page Optimization | | aj6130 -
Category page canonical tag
I know this question has been asked a few times on here but I'm looking for very specific advice. Currently when you go to a category, say http://www.bronterose.co.uk/range.html, a canonical tag is added to the head of the page. There are plenty of "variant" pages which carry the same tag, for example: /range.html?p=2
On-Page Optimization | | crichardson9
/range.html?p=3
/range.html?dir=asc&order=price
/range.html?dir=asc&limit=all&order=price Is it wise to push the "link juice" for each of these variant pages to the top level page? Or should each variant page have its own unique canonical tag? After reading many blog posts, guides and papers I'm truly confused! Any general guidance or recommendations would be much appreciated. Chris.1 -
Best SEO Extension/Plugin for NOPCommerce Site?
Hi I am working for a client who is using NOPCommerce. It doesn't look like they have a SEO Plugin in - although you can add meta descriptions to Products - which works fine, the Product categories have SEO components too but do not seem to work and all 'other' content /CMS pages have no SEO components whatsoever. Does anyone know of a plugin which would resolve this? (PS never used NOPCommerce before!)
On-Page Optimization | | AllieMc0 -
How to properly remove pages and a category from Google's index
I want to remove this category http://www.webdesign.org/web-design-news-all/ and all the pages in that category (e.g. http://www.webdesign.org/web-design-news-all/7386.html ) from Google's index. I used the following string in the "Reomval URS" section in Google Webmaster Tools: http://www.webdesign.org/web-design-news-all/* is that correct or I better use http://www.webdesign.org/web-design-news-all/ ? Thanks in advance.
On-Page Optimization | | VinceWicks0 -
How to avoid keyword stuffing on e-Commerce Category pages
Hi, I'm optimizing a large, consumer electronic e-commerce superstore. Based on client's choice of keywords, I'm using product category pages as my target urls. Because of the proprietary CMS structure, product names and titles, featured on my landing pages (product category pages) create a keyword overkill, affecting various ranking factors. For example, one of the target urls / landing pages, dedicated to a specific product category, mentions the keyword over 190 times because of so many product titles in the "body" section. Would inline "rel="canonical" help? If yes, what part of the website should it "canonize"? If rel="canonical" is not the answer, what strategies would you suggest? Thanks!
On-Page Optimization | | dimanyc0 -
Title tag for category page
I'd like to know your views on the best approach for title tags for category pages for ecommerce sites. 3 examples A) Category name | Free delivery on $50 purchase | Brand name B) Discover best "category name" on brand name C) Category Name | 1st Keyword, 2nd keyword | Brand name Thanks!
On-Page Optimization | | walidalsaqqaf0 -
Impact of removing category sidebar with keywords?
Our site (a niche financial publication: insideARM.com) requires some more room in the sidebar. We're considering removing the categories (we call them topics) sidebar block, or cutting down the number of items displayed within it. My concern is that we'd be removing a direct link to landing pages for important keyword terms from our most powerful page (the index). Sure, we have the terms listed in the footer, but I am worried that the position change will lower the value of the links. Our users don't really use these links for navigational purposes, which is why it comes up as a potential removed item. Am I wrong to worry about this? Would we be crippling our category pages by doing this?
On-Page Optimization | | insideARM0