How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Eliminate render-blocking CSS in above the fold content with shopify
Hello, It seems that my shopify website needs to eliminate render-blocking CSS in above the fold content to get better mobile speed and I don't see or find anything about how this could be done. I have succeeded with a wordpress site but the way things are restricted with shopify, I have not been able to do the same. The other thing would be to leverage browser caching which I have no clue how it can be done with shopify either. Thanks!
On-Page Optimization | | Mabo320 -
Will it upset Google if I aggregate product page reviews up into a product category page?
We have reviews on our product pages and we are considering averaging those reviews out and putting them on specific category pages in order for the average product ratings to be displayed in search results. Each averaged category review would be only for the products within it's category, and all reviews are from users of the site, no 3rd party reviews. For example, averaging the reviews from all of our boxes products pages, and listing that average review on the boxes category page. My question is, will this be doing anything wrong in the eyes of Google, and if so how so? -Derick
On-Page Optimization | | Deluxe0 -
Can you use the canonical tag and rel=next and rel=prev on category pages.
We have a conflict of information between our web developers and our SEO company. We are an on-line retail company hence we have a fair number of different categories. Our site is set up with the rel=next and rel=prev tags. Our SEO company have asked us to implement canonical links on our category pages and leave the rel=next and rel=prev tags as they are. Our web developers are saying by doing this we are asking Google to ignore all of our products on all of the pages except page 1 which would mean Google would not index a lot of our products. I have looked at a few articles but I am struggling to understand which way to go. Any advice would be appreciated. Thank you in advance.
On-Page Optimization | | Palmbourne0 -
Keyword/phrase proximity
I'm curious about opinions regarding how the search algorithms treat multiple key phrases that may reside in one long tail key phrase. So for example: If I'm optimizing for "New York Litigation Lawyer", would that also give me rankings for "New York Lawyer"? My thought is that the former will be considered the primary keyword and rankings will improve mostly for that, but that the latter keyword could also possibly see some lift as well. Thanks!
On-Page Optimization | | nickturner9221 -
Www.colourbanners.co.uk/ & colourbanners.co.uk showing up as two seperate URLs - is this going to be dupliacte content issue?
Hi Guys, I have just created a report in Moz and there appears to be 91 duplicate content issues with the site which i need to fix as i think it could be the reason why we are suffering from a penalty. One of the main questions i have is these 3 variations of the URL http://www.colourbanners.co.uk/ http://colourbanners.co.ukhttp://colourbanners.co.uk/Each have links pointing to them. My question is, could this be causing a dupe issue?regardsGerry
On-Page Optimization | | gezzagregz0 -
Ecommerce- Keyword use in Product links on Category page
I'm wondering how Keyword use in Product links on Category pages can affect a pages rank? I have 1 site where this seems to be an issue but not on all categories. For this site, a site: keyword search ranks the category page as no.1 in the SERPS but a non-site: search shows 1 of the many products within the category as the highest ranking page (currently 20 in google) on this site. This product is probably the least likely to generate a conversion due to it's cost so this is less than ideal. The plural search of the keyword shows the category page and it ranks higher than the keyword itself (currently 9 in google) Category name and URL = keyword. The category is paginated with 12 products per page. Product URL and anchor text is brand-model-type (where type = keyword) I'd like to keep the product URLs and anchors as they are if I can as they are well searched terms themselves but I want to optimize a category page to rank for the keyword itself. Have any of you overcome a similar issue? Would adding more text to the category page dilute the issue?
On-Page Optimization | | MarcOZ0 -
Yahoo Directory! Is it worth Paying $299 for?
Yahoo Directory! Is it worth Paying $299 for? Is this going to help you be better indexed. Will it help your site over all beside having one link?
On-Page Optimization | | AppleCapitalGroup0 -
How do we handle sitemaps in robots.txt when multiple domains point to same physical location?
we have www.mysite.net, www.mysite.se, www.mysite.fi and so on. all of these domains point to the same physical location on our webserver, and we replace texts given back to client depending on which domain he/she requested. My problem is this: How do i configure sitemaps in robots.txt when robots.txt is used by multiple domains? If I for instance put the rows Sitemap: http://www.mysite.net/sitemapNet.xml
On-Page Optimization | | nordicnetproducts
Sitemap: http://www.mysite.net/sitemapSe.xml in robots.txt, would that result in some cross submission error?0