How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Wordpress / Full URL In Menu Box
I came across an article online (not Moz) that says adding the full url in a menu is a Google standard. So when you make a menu link you put "www.example.com/page" instead of "/page". What are your thoughts on this? Any real reason to? Y840pbN lrwZPDj
On-Page Optimization | | InfinityTechnologySolutions0 -
Using Robots Meta Tag on Review Form Pages
I have gone over this so many times and I just can't seem to get it straight and hope someone can help me out with a couple of questions: Right now, on my dynamically created pages created by filters (located on the category pages) I am using rel""canonical" to point them to their respective category page. Should I also use the robots meta tag as well? Similarly, each product I have on my site has a review form on it and thus is getting indexed by Google. I have placed the same canonical tag on them as well pointing them to the page with the review form on it. In the past I used robots.txt to block google from the review pages but this didn't really do much. Should I be using the robots meta tag on these pages as well? If I used the robots meta tag should I noindex,nofollow? Thanks in advance, Jake
On-Page Optimization | | jake3720 -
Keep our category navigation in tree structure but move our URLs to a more flat structure. Good plan?
In our Magento store, products are arranged into categories, subcategories and so on. We typically have 3 or 4 layers of category depth. This makes it nice and easy for customers to find stuff, but it means that the end categories have massive long urls. I'd like to keep our category tree structure in place from a navigation point of view, but I feel the url structure is pushing some important stuff to the back of the shop as it were. We have something like 200 categories in total. So, assuming every individual category has an a unique name, I'd like to rewrite the urls so that: ourshop.com/car-parts/
On-Page Optimization | | DWJames
stays as
ourshop.com/car-parts/ ourshop.com/car-parts/suspension/
becomes
ourshop.com/suspension/ ourshop.com/car-parts/suspension/springs
becomes
ourshop.com/springs/ ourshop.com/car-parts/suspension/springs/thismake-lowering-springs
becomes
ourshop.com/thismake-lowering-springs/ and so on....
I'll need some custom magento URL rewrite work done, but that's another story. The real question is whether you guys feel this is worthwhile? Are there any other stores with a deep categorised navigation structure, but a flat url structure? thanks, James0 -
A / B Split testing
If you have developed a server-side A / B split testing tool. How do you avoid that Google will think you are doing cloaking. When a user enters www.xxxxx.com/xxxxxx/ he gets either version 1 or version 2 back from the server on the same url. If the user does not accept session cookies he will always get version 1.
On-Page Optimization | | Stener0 -
Best practice for Meta-Robots tag in categories and author pages?
For some of our site we use Wordpress, which we really like working with. The question I have is for the categories and authors pages (and similiar pages), i.e. the one looking: http://www.domain.com/authors/. Should you or should you not use follow, noindex for meta-robots? We have a lot of categories/tags/authors which generates a lot of pages. I'm a bit worried that google won't like this and leaning towards adding the follow, noindex. But the more I read about it, the more I see people disagree. What does the community of Seomoz think?
On-Page Optimization | | Lobtec0 -
Why does http://www.southernhearth.com rank so well for patio furniture and outdoor furniture?
I was looking at the competitiveness of ranking for the keywords "patio furniture" & "outdoor furniture" and was shocked to see www.southernhearth.com ranking so well. While I am just getting into the research, I was unable to see anything that would justify it's rank. What are your thoughts?
On-Page Optimization | | Mattbyo0 -
Title tag for category page
I’d love some clarification on what would be the best title tag for a shopping category page. The category page is “prams” , the brand is Baby Huddle and the top keywords in order of importance are pushchairs, strollers and buggies. Here are the options I suggest: Buy Prams | Pushchairs, Strollers, Buggies | Baby Huddle Buy the best prams, pushchairs, strollers and buggies on Baby Huddle Buy prams with free delivery and great prices on Baby Huddle
On-Page Optimization | | walidalsaqqaf0 -
How do you create a 301 redirect for www.mysite..com/index.html in htaccess.
I understand that it is possible to create a 301 redirect for www.mysite..com/index.html to www.mysite.com. (as well as subdirectories.) How is this accomplished? My hosting company says that setting this up in htaccess will cause "Apache to geti into an infinite loop and the page won’t load." I have read on the forum that this is possible. Any help would be greatly appreaciated. THanks. Perri
On-Page Optimization | | PerriCline0