Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Impact of keyword/keyphrases density on header/footer
Hi, It might be a stupid question but I prefer to clear things out if it's not a problem: Today I've seen a website where visitors are prompted no less than 5 times per page to "call [their] consultants".
On-Page Optimization | | GhillC
This appears twice on the header, once on the side bar (mouse over pop up), once in the body of most of the pages and once in the footer. So obviously, besides the body of the pages, it appears at least 4 times on every single pages as it's part of the website template. In the past, I never really wondered re the menu, the footer etc as it's usually not hammering the same stuff repeatedly everywhere. Anyway, I then had a look at their blog and, given the average length of their articles, the keyword density around these prompts is about 0.5% to 0.8% for each page. This is huge! So basically my question is as follow: is Google's algorithm smart enough to understand what this is and make abstraction of this "content" to focus on the body of the pages (probably simply focusing on the tags)? Or does it send wrong signals and confuse search engine more than anything else? Reading stuff such as this, I wonder how does it work when this is not navigational or links elements. Thanks,
G Note: I’m purposely not speaking about the UX which is obviously impacted by such a hammering process.0 -
Will it upset Google if I aggregate product page reviews up into a product category page?
We have reviews on our product pages and we are considering averaging those reviews out and putting them on specific category pages in order for the average product ratings to be displayed in search results. Each averaged category review would be only for the products within it's category, and all reviews are from users of the site, no 3rd party reviews. For example, averaging the reviews from all of our boxes products pages, and listing that average review on the boxes category page. My question is, will this be doing anything wrong in the eyes of Google, and if so how so? -Derick
On-Page Optimization | | Deluxe0 -
SERP Hijacking/Content Theft/ 302 Redirect?
Sorry for the second post, thought this should have it's own. Here is the problem I am facing amongst many others. Let's take the search term "Air Jordan Release Dates 2017" and place it into Google Search. Here is a link:
On-Page Optimization | | SneakerFiles
https://www.google.com/#q=air+jordan+release+dates+2017 Towards the bottom of the page, you will see a website that has SneakerFiles (my website) in the title. The exact title is: Air Jordan Release Dates 2016, 2017 | SneakerFiles - Osce Now, this is my content, but not my website. For some reason, Google thinks this is my site. If you click on the link in search, it automatically redirects you to another page (maybe 302 redirect), but in the cache you can see it's mine:
http://webcache.googleusercontent.com/search?q=cache:qrVEUDE1t48J:www.osce.gob.pe/take_p_firm.asp%3F+&cd=8&hl=en&ct=clnk&gl=us I have blocked the websites IP, disallowed my style.css to be used so it just shows a links without the style, still nothing. I have submitted multiple google spam reports as well as feedback from search. At times, my page will return to the search but it gets replaced by this website. I even filed a DMCA with Google, they declined it. I reached out to their Host and Domain register multiple times, never got a response. The sad part about this, it's happening for other keywords, for example if you search "KD 9 Colorways", the first result is for my website but on another domain name (my website does rank 3rd for a different Tag page). The page I worked hard on keeping up to date. I did notice this bit of javascript from the cloaked/hacked/serp hijacking website: I disabled iFrames...(think this helps) so not sure how they are doing this. Any help would be greatly appreciated. Note: I am using Wordpress if that means anything.0 -
Why are http and https pages showing different domain/page authorities?
My website www.aquatell.com was recently moved to the Shopify platform. We chose to use the http domain, because we didn't want to change too much, too quickly by moving to https. Only our shopping cart is using https protocol. We noticed however, that https versions of our non-cart pages were being indexed, so we created canonical tags to point the https version of a page to the http version. What's got me puzzled though, is when I use open site explorer to look at domain/page authority values, I get different scores for the http vs. https version. And the https version is always better. Example: http://www.aquatell.com DA = 21 and https://www.aquatell.com DA = 27. Can somebody please help me make sense of this? Thanks,
On-Page Optimization | | Aquatell1 -
Removing navigation menu items/links on homepage
We are redesigning our website after a long stint with an SEO firm who also handled our design/dev. We want to clean up the links on our homepage but don't want to screw up our IA or SEO. We want to delete some navbar menu items and a whole bunch on random links to our evergreen content below the fold. Would we need to reposition those navbar items/content links to our footer or somewhere else on the homepage to maintain our internal linking structure? It would be great if you could take a look at our site and give us any suggestions or advice on the best way to go about this. Thanks!
On-Page Optimization | | Lorne_Marr1 -
No-index all the posts of a category
Hi everyone! I would like no-indexing all the posts of a specific category of my wordpress site. The problem is that the structure of my URL is composed without /category/: www.site-name.ext/date/post-name/
On-Page Optimization | | salvyy
so without /category-name/ Is possibile to disallow the indexing of all the posts of the category via robots.txt? Using Yoast Plugin I can put the no-index for each post, but I would like to put the no-index (or disallow/) a time for all the post of the category. Thanks in advance for your help and sorry for my english. Mike0 -
Right way to block google robots from ppc landing pages
What is the right way to completely block seo robots from my adword landing pages? Robots.txt does not work really good for that, as far I know. Adding metatags noindex nofollow on the other side will block adwords robot as well. right? Thank you very much, Serge
On-Page Optimization | | Kotkov0 -
Category Pages with Sub-Categories
The image will explain it all... Each category page starts on the subject of the first sub-category page. This happens twice (well actually 3 times since this section of the site is called showroom and it starts on the tab mowers). Is this a terrible approach? If so, how could a site like this be better navigation-ally organized. cat-subcat.png
On-Page Optimization | | drewschmaltz0