How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I robots.txt an entire site to get rid of Duplicate content?
I am in the process of implementing Zendesk and will have two separate Zendesk sites with the same content to serve two separate user groups (for the same product-- B2B and B2C). Zendesk does not allow me the option to changed canonicals (nor meta tags). If I robots.txt one of the Zendesk sites, will that cover me for duplicate content with Google? Is that a good option? Is there a better option. I will also have to change some of the canonicals on my site (mysite.com) to use the zendesk canonicals (zendesk.mysite.com) to avoid duplicate content. Will I lose ranking by changing the established page canonicals on my site go to the new subdomain (only option offered through Zendesk)? Thank you.
On-Page Optimization | | RoxBrock0 -
No meta description on category page
Hi Moz is reporting no meta description on a wordpress category page like this one: http://www.dwliverpoolphotography.co.uk/category/uncategorized/ Can I add a meta description to a category? Best wishes. David.
On-Page Optimization | | WallerD0 -
Blocking Subdomain from Google Crawl and Index
Hey everybody, how is it going? I have a simple question, that i need answered. I have a main domain, lets call it domain.com. Recently our company will launch a series of promotions for which we will use cname subdomains, i.e try.domain.com, or buy.domain.com. They will serve a commercial objective, nothing more. What is the best way to block such domains from being indexed in Google, also from counting as a subdomain from the domain.com. Robots.txt, No-follow, etc? Hope to hear from you, Best Regards,
On-Page Optimization | | JesusD3 -
Need suggestion: Should the user profile link be disallowed in robots.txt
I maintain a myBB based forum here. The user profile links look something like this http://www.learnqtp.com/forums/User-Ankur Now in my GWT, I can see many 404 errors for user profile links. This is primarily because we have tight control over spam and auto-profiles generated by bots. Either our moderators or our spam control software delete such spammy member profiles on a periodic basis but by then Google indexes those profiles. I am wondering, would it be a good idea to disallow User profiles links using robots.txt? Something like Disallow: /forums/User-*
On-Page Optimization | | AnkurJ0 -
Looking for a few hours of consult from an on-page/redirection SEO guru.
Hi! I could post our question here, but I think we will need an hour or so of questions and answers. Anyone consider themselves an on-page guru or redirection strategy guru? Here is the breakdown: 1. We sell digital media. 2. Most of the pages that return for us in the serps are indexed Search Results pages. From the research we have done, these are our highest trafficked results, but also our lowest converters. We have hundreds of thousands of indexed and dynamic Search Result pages we need to deal with. 3. We are concerned about Panda eventually giving us issues with these indexed Search Pages. 4. We would like to 301 re-direct these indexed Search Results pages to applicable product pages. 5. Looking for advice on a strategy to do so which would include the best way to locate the pages we need to 301 in the SERPS, being careful that the 301 will help and not harm us, etc. Note, we are just looking for a couple of hours of discussion and guidance. We don't need a massive SEO analysis or anything. Just looking for input and guidance on this one particular issue. If you just want to answer here for free, hey, we won't complain, but are happy to pay for quality advice. PM if interested. Thanks! Craig
On-Page Optimization | | TheCraig0 -
How To Get GD Star Rating working in wordpress categories.
I want to use GD star rating so that stars show up in SERPS for my category pages. I've installed the plugin but can't see how to do it. Does anyone have any idea?
On-Page Optimization | | SamCUK0 -
Which is better, a directory 301 redirect or each page in the directory?
A customer of mine has a site with lots of articles and they are all quite spammy. They have not been affected by penguin yet so they asked what to do. I suggested losing the articles directory and 301 redirect to either the home page or another important page. Would a 301 redirect on the entire directory to a single page be the way to go or add redirects from each page within the directory and spread out redirects to various pages in website? Or do you have a better suggestion?
On-Page Optimization | | anthonytjm0 -
SEF URLs. Should I use / or - ?
I have o activate SEF URLs in a website. Regarding SEO, is there any difference between using / or - ? I mean, Is it better to write URLs like this: http://www.domain.com/folder/folder/page or like this: http://www.domain.com/folder-folder-page ? Is there any difference? Thanks!
On-Page Optimization | | ociosu0