Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I replace categories with a static page
Hello there. I want to replace all of WordPress categories with static pages so that users see a well designed and constructed presentation of all the articles within each topic instead of just a long list of excerpts. I've already done this with 2 categories and although it is hard work I can't help feeling it is a much better thing for my users. However, I'm concerned that I am embarking on this project without being totally sure that it makes sense from an Seo point of view, or whether there are any downsides I haven't thought of? My idea is that the WordPress categories are set to noindex and nofollow. Search engines should find all of my static category pages and all of the content within each category will be spidered from there instead. Just to be sure you know what I mean here is a link to a normal category - https://www.whitegoodshelp.co.uk/category/consumer/ and here is my static page replacement for it - https://www.whitegoodshelp.co.uk/consumer-rights-appliances/ Both pages contain links to all articles within the category except the one generated by WordPress is just a long paginated list, and my replacement is a proper category page, which is hopefully far more useful . Can someone please confirm that there are no downsides to this strategy? 🙂
On-Page Optimization | | Snowdune1 -
SERP Hijacking/Content Theft/ 302 Redirect?
Sorry for the second post, thought this should have it's own. Here is the problem I am facing amongst many others. Let's take the search term "Air Jordan Release Dates 2017" and place it into Google Search. Here is a link:
On-Page Optimization | | SneakerFiles
https://www.google.com/#q=air+jordan+release+dates+2017 Towards the bottom of the page, you will see a website that has SneakerFiles (my website) in the title. The exact title is: Air Jordan Release Dates 2016, 2017 | SneakerFiles - Osce Now, this is my content, but not my website. For some reason, Google thinks this is my site. If you click on the link in search, it automatically redirects you to another page (maybe 302 redirect), but in the cache you can see it's mine:
http://webcache.googleusercontent.com/search?q=cache:qrVEUDE1t48J:www.osce.gob.pe/take_p_firm.asp%3F+&cd=8&hl=en&ct=clnk&gl=us I have blocked the websites IP, disallowed my style.css to be used so it just shows a links without the style, still nothing. I have submitted multiple google spam reports as well as feedback from search. At times, my page will return to the search but it gets replaced by this website. I even filed a DMCA with Google, they declined it. I reached out to their Host and Domain register multiple times, never got a response. The sad part about this, it's happening for other keywords, for example if you search "KD 9 Colorways", the first result is for my website but on another domain name (my website does rank 3rd for a different Tag page). The page I worked hard on keeping up to date. I did notice this bit of javascript from the cloaked/hacked/serp hijacking website: I disabled iFrames...(think this helps) so not sure how they are doing this. Any help would be greatly appreciated. Note: I am using Wordpress if that means anything.0 -
Using keywords in my URL: Doing a redirect to /keyword
My website in "On Page Grade" received an A.Anyway, I only have 1 thing to optimize:_"Use Keywords in your URL__Using your targeted keywords in the URL string adds relevancy to your page for search engine rankings, assists potential visitors identify the topic of your page from the URL, and provides SEO value when used as the anchor text of referring links."_My website is ranking in top10 for a super high competitive keyword and all my others competitors have the keyword on their domain, but not for my URL.Since I can't change my domain for fixing this suggestion, I would like to know what do you think about doing a 301 redirect from / to mydomainname.com/keyword/So the index of my website would be the /keyword.I don't know if this can make a damage to my SERP for the big change ir it would be a great choice.
On-Page Optimization | | estebanseo0 -
No-index all the posts of a category
Hi everyone! I would like no-indexing all the posts of a specific category of my wordpress site. The problem is that the structure of my URL is composed without /category/: www.site-name.ext/date/post-name/
On-Page Optimization | | salvyy
so without /category-name/ Is possibile to disallow the indexing of all the posts of the category via robots.txt? Using Yoast Plugin I can put the no-index for each post, but I would like to put the no-index (or disallow/) a time for all the post of the category. Thanks in advance for your help and sorry for my english. Mike0 -
Best practice for footer in ecommerce - Shall I add Top Category links?
What would you recommend regarding links to "Top Products" and "Top Categories" in footer? Would you add them to give extra link juice to top categories? would you try to avoid category links in footer that are already in the header navigationor in the main content area to avoid linking twice from all pages? would you vary these top category links in footer according to main category
On-Page Optimization | | lcourse0 -
How To Change Wordpress Category Title
My categories are indexed and I want to change the category page title. At present it just defaults to the category name but I want to set a different page title. For example I want the category to be 'Motor Cars' but I want the category page title to be 'Buy Motor Cars - New And Used'. How can I do this?
On-Page Optimization | | SamCUK0 -
Does show/hide element with javascript impact SEO
Hi I am developing an ecommerce site and want to place text on all category and home page. The challenge is that 300 words of text for the pages does not fit into the design appropriately especially on the home page. If I were to use a show/hide element with javascript would this be seen as spam or a trick to the search engines. I do not think it is spam as it will be actual content for the site and the visitor can view it if they click on the show button. Would love to hear your thoughts?
On-Page Optimization | | VivaArturo0 -
Right way to block google robots from ppc landing pages
What is the right way to completely block seo robots from my adword landing pages? Robots.txt does not work really good for that, as far I know. Adding metatags noindex nofollow on the other side will block adwords robot as well. right? Thank you very much, Serge
On-Page Optimization | | Kotkov0