Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Updating Old Content at Scale - Any Danger from a Google Penalty/Spam Perspective?
We've read a lot about the power of updating old content (making it more relevant for today, finding other ways to add value to it) and republishing (Here I mean changing the publish date from the original publish date to today's date - not publishing on other sites). I'm wondering if there is any danger of doing this at scale (designating a few months out of the year where we don't publish brand-new content but instead focus on taking our old blog posts, updating them, and changing the publish date - ~15 posts/month). We have a huge archive of old posts we believe we can add value to and publish anew to benefit our community/organic traffic visitors. It seems like we could add a lot of value to readers by doing this, but I'm a little worried this might somehow be seen by Google as manipulative/spammy/something that could otherwise get us in trouble. Does anyone have experience doing this or have thoughts on whether this might somehow be dangerous to do? Thanks Moz community!
On-Page Optimization | | paulz9990 -
Why are http and https pages showing different domain/page authorities?
My website www.aquatell.com was recently moved to the Shopify platform. We chose to use the http domain, because we didn't want to change too much, too quickly by moving to https. Only our shopping cart is using https protocol. We noticed however, that https versions of our non-cart pages were being indexed, so we created canonical tags to point the https version of a page to the http version. What's got me puzzled though, is when I use open site explorer to look at domain/page authority values, I get different scores for the http vs. https version. And the https version is always better. Example: http://www.aquatell.com DA = 21 and https://www.aquatell.com DA = 27. Can somebody please help me make sense of this? Thanks,
On-Page Optimization | | Aquatell1 -
Best practice for footer in ecommerce - Shall I add Top Category links?
What would you recommend regarding links to "Top Products" and "Top Categories" in footer? Would you add them to give extra link juice to top categories? would you try to avoid category links in footer that are already in the header navigationor in the main content area to avoid linking twice from all pages? would you vary these top category links in footer according to main category
On-Page Optimization | | lcourse0 -
Category page canonical tag
I know this question has been asked a few times on here but I'm looking for very specific advice. Currently when you go to a category, say http://www.bronterose.co.uk/range.html, a canonical tag is added to the head of the page. There are plenty of "variant" pages which carry the same tag, for example: /range.html?p=2
On-Page Optimization | | crichardson9
/range.html?p=3
/range.html?dir=asc&order=price
/range.html?dir=asc&limit=all&order=price Is it wise to push the "link juice" for each of these variant pages to the top level page? Or should each variant page have its own unique canonical tag? After reading many blog posts, guides and papers I'm truly confused! Any general guidance or recommendations would be much appreciated. Chris.1 -
Do Parent Categories Hurt SEO?
I have parent categories and subcategories. Will it be harder for the subcategories to rank well because they have a parent category? The URL is longer, for one. I am just wondering if I should not have parent categories. I have one category page doing really well and I am trying to boost the others (most of which are subcategories) and this is a concern for me. Thanks! Edit: I also have a category that has 2 parent categories. I want it automatically in those 2 categories and one of its own. By itself it is very important keyword. Is this ok or should I have it be a parent category?
On-Page Optimization | | 2bloggers0 -
Ecommerce - how many clicks from the home page should categories be
My client has about 300 products in 20 categories with a lot of overlap. How many clicks from the home page should we keep the products? We're not doing pagination. I'd been told several years ago that all products should be 2 clicks or less from the home page. Is this true today? Thanks.
On-Page Optimization | | BobGW1 -
Http://www.xxxx.com does not re-direct to http://xxx.com
When typing in my website URL www.earthsaverequipment.com successfully re-directs to earthsaverequipment.com as specified in robot. However if you type http://www.earthsaverequipment.com it brings up a 404 error Is this a potential issue? if so is there a way to fix it? thanks
On-Page Optimization | | Earthsaver0 -
How do we handle sitemaps in robots.txt when multiple domains point to same physical location?
we have www.mysite.net, www.mysite.se, www.mysite.fi and so on. all of these domains point to the same physical location on our webserver, and we replace texts given back to client depending on which domain he/she requested. My problem is this: How do i configure sitemaps in robots.txt when robots.txt is used by multiple domains? If I for instance put the rows Sitemap: http://www.mysite.net/sitemapNet.xml
On-Page Optimization | | nordicnetproducts
Sitemap: http://www.mysite.net/sitemapSe.xml in robots.txt, would that result in some cross submission error?0