Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I replace categories with a static page
Hello there. I want to replace all of WordPress categories with static pages so that users see a well designed and constructed presentation of all the articles within each topic instead of just a long list of excerpts. I've already done this with 2 categories and although it is hard work I can't help feeling it is a much better thing for my users. However, I'm concerned that I am embarking on this project without being totally sure that it makes sense from an Seo point of view, or whether there are any downsides I haven't thought of? My idea is that the WordPress categories are set to noindex and nofollow. Search engines should find all of my static category pages and all of the content within each category will be spidered from there instead. Just to be sure you know what I mean here is a link to a normal category - https://www.whitegoodshelp.co.uk/category/consumer/ and here is my static page replacement for it - https://www.whitegoodshelp.co.uk/consumer-rights-appliances/ Both pages contain links to all articles within the category except the one generated by WordPress is just a long paginated list, and my replacement is a proper category page, which is hopefully far more useful . Can someone please confirm that there are no downsides to this strategy? 🙂
On-Page Optimization | | Snowdune1 -
Using h2 for category on ecommerce website
Hi, I am working on an ecommerce site and the main category - lets call them car widgets - is using a h1 at the top of the page which is great. There are 4 sub categories on the page - lets call one of them red widget. The only content on the page is the sub category name and an image. Should the sub category red widget use a h2? Thanks S
On-Page Optimization | | bedynamic0 -
Should we rename and update a page or create a new page entirely?
Hi Moz Peoples! We have a small site with a simple site navigation, with only a few links on the nav bar. We have been doing some work to create a new page, which will eventually replace one of the links on the nav bar. The question we are having is, is it better to rename the existing page and replace its content and then wait for the great indexer to do its thing, or perm delete the page and replace it with the new page and content? Or is this a case where it really makes no difference as long as the redirects are set up correctly?
On-Page Optimization | | Parker8180 -
Alt text / internal linking
Hi everyone A question about best practice when linking from pictures on our homepage - hirespace.com We have an option of using divs with background images (nicer in terms of design) but it means that we can't use anchor text or alt text to show Google what these internal links are about. The other option is to use images which do not allow us as much flexibility in terms of CSS but would allow us to use alt text. There is also an opinion that we should have separate text links at the bottom of the homepage to get the anchor page in. What is best practice in this situation - is alt text worth sacrificing some CSS flexibility for? How important is anchor/alt text for internal linking? Thanks guys.
On-Page Optimization | | HireSpace0 -
Best practice for footer in ecommerce - Shall I add Top Category links?
What would you recommend regarding links to "Top Products" and "Top Categories" in footer? Would you add them to give extra link juice to top categories? would you try to avoid category links in footer that are already in the header navigationor in the main content area to avoid linking twice from all pages? would you vary these top category links in footer according to main category
On-Page Optimization | | lcourse0 -
How To Change Wordpress Category Title
My categories are indexed and I want to change the category page title. At present it just defaults to the category name but I want to set a different page title. For example I want the category to be 'Motor Cars' but I want the category page title to be 'Buy Motor Cars - New And Used'. How can I do this?
On-Page Optimization | | SamCUK0 -
WordPress and category/subcategory landing pages
Hey, Here's my situation. I'm building a WordPress blog for product reviews of a certain niche. Current category setup is 4 main categories with 4-8 subcategories each. Each subcategory has a unique description that will help it become a landing page for certain keywords, after which it lists the posts from that subcategory. The posts will always be assigned to a sub-category, never to a main category. My issue is what to do with the main categories. They're fairly general so they're not really targeting any keywords, and don't have any unique descriptions attached to them. I was thinking of choosing between three options on designing the main category pages: List the subcategories + normal posts loop that bring the latest posts from the subcategories (may create a lot of duplicate content since the subcategory pages are also listing their posts) List only the subcategories (+ maybe just the latest post from each subcategory) Don't link the main categories at all, instead only use them to create dropdowns for the subcategories So, what would you choose, and why?
On-Page Optimization | | mihaiaperghis0 -
Title tag for category page
I'd like to know your views on the best approach for title tags for category pages for ecommerce sites. 3 examples A) Category name | Free delivery on $50 purchase | Brand name B) Discover best "category name" on brand name C) Category Name | 1st Keyword, 2nd keyword | Brand name Thanks!
On-Page Optimization | | walidalsaqqaf0