Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does DA/PA have any effect on rankings?
I have seen many people are concerned about increasing DA and PA of their websites. While I am very curious why do people focus on increasing DA and PA? Does DA and PA effect the rankings of the website? Because I have recently launched my website regarding men beard trimmer and it is ranking on 1st page but not on number 1 position. Will increasing DA/PA of the site help me in occupying 1st position?
On-Page Optimization | | RyanAmin0 -
How to exclude URL filter searches in robots.txt
When I look through my MOZ reports I can see it's included 'pages' which it shouldn't have included i.e. adding filtering rules such as this one http://www.mydomain.com/brands?color=364&manufacturer=505 How can I exclude all of these filters in the robots.txt? I think it'll be: Disallow: /*?color=$ Is that the correct syntax with the $ sign in it? Thanks!
On-Page Optimization | | neenor0 -
Best SEO Extension/Plugin for NOPCommerce Site?
Hi I am working for a client who is using NOPCommerce. It doesn't look like they have a SEO Plugin in - although you can add meta descriptions to Products - which works fine, the Product categories have SEO components too but do not seem to work and all 'other' content /CMS pages have no SEO components whatsoever. Does anyone know of a plugin which would resolve this? (PS never used NOPCommerce before!)
On-Page Optimization | | AllieMc0 -
How much copy should there be on an e-commerce category page?
I'm not looking for a precise number, obviously. I'm more interested in a general range. More text means more long-tail and synonym opportunities, but of course you don't want too much copy above the fold, pushing your products down. Maybe you can get away with a short paragraph or two at the top of the page. You can always put more copy below the products, but in a recent SEOmoz e-commerce webinar, the presenter seemed to think that was silly and unnecessary. He even suggested that the algo might intentionally ignore text below products, since it's clearly not intended to be read. What do you think?
On-Page Optimization | | CMC-SD0 -
Does show/hide element with javascript impact SEO
Hi I am developing an ecommerce site and want to place text on all category and home page. The challenge is that 300 words of text for the pages does not fit into the design appropriately especially on the home page. If I were to use a show/hide element with javascript would this be seen as spam or a trick to the search engines. I do not think it is spam as it will be actual content for the site and the visitor can view it if they click on the show button. Would love to hear your thoughts?
On-Page Optimization | | VivaArturo0 -
Http://www.xxxx.com does not re-direct to http://xxx.com
When typing in my website URL www.earthsaverequipment.com successfully re-directs to earthsaverequipment.com as specified in robot. However if you type http://www.earthsaverequipment.com it brings up a 404 error Is this a potential issue? if so is there a way to fix it? thanks
On-Page Optimization | | Earthsaver0 -
Best practice for Meta-Robots tag in categories and author pages?
For some of our site we use Wordpress, which we really like working with. The question I have is for the categories and authors pages (and similiar pages), i.e. the one looking: http://www.domain.com/authors/. Should you or should you not use follow, noindex for meta-robots? We have a lot of categories/tags/authors which generates a lot of pages. I'm a bit worried that google won't like this and leaning towards adding the follow, noindex. But the more I read about it, the more I see people disagree. What does the community of Seomoz think?
On-Page Optimization | | Lobtec0 -
Wordpress: Should I NO INDEX Categories & Archives Pages?
I am new to SEOmoz & trying to work my way through the ca-trillion errors that have been found on my site, but for each one I want to ensure that I am helping rather than harming my site. The tool has (as a "notice") said that my category pages & Archives are NO-INDEX, is this how these pages should be dealt with? In addition, the crawler has also (as a "warning error) discovered that my categories, and Archives do not have a meta description..is this of great importance for non indexed pages of this type? Thanks so much to the SEOmoz forum members, you have so far been of invaluable help to me.
On-Page Optimization | | KMack2