Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I outrank a website whose brand is named specifically after a product / service (and mine is not!) ?
Hi everyone, I've been working hard on my on-page SEO lately so I can gain visibility for my website. The results have been great and I am now on top of the SERP for the services I propose. I think that my content is almost fully optimize and** I've respected all the best practices (both on-page and technical SEO)**. However, there is one problem I just can't deal with for I don't have the knowledge, hence this post. I'm pretty sure that this issue is quite common for SEO experts. Here the thing: I offer dental emergency services, which is my core business. I'm ranked 4th for that request in my area, which is good, but I'd like to rank 1st as** I have a better DA and content than the 3 websites outranking me**. Also, I'm the first result for any other related services such as "dental services" or "dentist". However, when it comes to the theme "dental emergency", I'm constantly outranked by the same 3 websites. I ran an audit on their website but **my content and technical SEO is way better **than theirs. I suppose that the only reason I'm behind them is because they used "dental emergency" in their **Brand name **and, therefore, in the Home page URL. Every time someone is looking up online for "dental emergency", these websites will be on top of the SERP as I think that Google is unable to know whether the users are specifically looking for their websites (aka Brand) or for "dental emergency" services. Here is an example of a competitor: https://www.urgencedentairedemontreal.com/ (urgence meaning emergency in French). His whole Brand name and URL have been built after the "dental emergency." service. On the contrary, **my Brand name does not mention "urgence". ** I see that as a trick that is confusing Google. The fact that my competitors named their Brand after a specific service I also offer is real pain for my SEO. I also think it's really unfair as I've put a lot of effort in designing a nice website with great UX and content. This is the kind of practice that should be penalized in my opinion. Please, does anyone know any way to resolve this issue?
On-Page Optimization | | AlexTL0 -
Category pages, should I noindex them?
Hi there, I have a question about my blog that I hope you guys can answer. Should I no index the category and tag pages of my blog? I understand they are considered as duplicate content, but what if I try to work the keyword of that category? What would you do? I am looking forward to reading your answers 🙂
On-Page Optimization | | lucywrites0 -
SERP Hijacking/Content Theft/ 302 Redirect?
Sorry for the second post, thought this should have it's own. Here is the problem I am facing amongst many others. Let's take the search term "Air Jordan Release Dates 2017" and place it into Google Search. Here is a link:
On-Page Optimization | | SneakerFiles
https://www.google.com/#q=air+jordan+release+dates+2017 Towards the bottom of the page, you will see a website that has SneakerFiles (my website) in the title. The exact title is: Air Jordan Release Dates 2016, 2017 | SneakerFiles - Osce Now, this is my content, but not my website. For some reason, Google thinks this is my site. If you click on the link in search, it automatically redirects you to another page (maybe 302 redirect), but in the cache you can see it's mine:
http://webcache.googleusercontent.com/search?q=cache:qrVEUDE1t48J:www.osce.gob.pe/take_p_firm.asp%3F+&cd=8&hl=en&ct=clnk&gl=us I have blocked the websites IP, disallowed my style.css to be used so it just shows a links without the style, still nothing. I have submitted multiple google spam reports as well as feedback from search. At times, my page will return to the search but it gets replaced by this website. I even filed a DMCA with Google, they declined it. I reached out to their Host and Domain register multiple times, never got a response. The sad part about this, it's happening for other keywords, for example if you search "KD 9 Colorways", the first result is for my website but on another domain name (my website does rank 3rd for a different Tag page). The page I worked hard on keeping up to date. I did notice this bit of javascript from the cloaked/hacked/serp hijacking website: I disabled iFrames...(think this helps) so not sure how they are doing this. Any help would be greatly appreciated. Note: I am using Wordpress if that means anything.0 -
Can we change Title and/or Descriptions Dynamically Based on Search Query
If I recall we used to be able to change our title attributes tag dynamically based on the search query but not sure if it's possible now or if it makes sense to do so. Thoughts? Rosemary
On-Page Optimization | | RosemaryB1 -
Solve duplicate content issues by using robots.txt
Hi, I have a primary website and beside that I also have some secondary websites with have same contents with primary website. This lead to duplicate content errors. Because of having many URL duplicate contents, so I want to use the robots.txt file to prevent google index the secondary websites to fix the duplicate content issue. Is it ok? Thank for any help!
On-Page Optimization | | JohnHuynh0 -
What is the right schema.org link for a web design / developer / mobile agency?
It seems strange that a group of web developers would make up an entire structured language to designate businesses by category and somehow forget to include companies like.... web developers. So I must be missing it, what is correct to use?
On-Page Optimization | | yeagerd0 -
Does show/hide element with javascript impact SEO
Hi I am developing an ecommerce site and want to place text on all category and home page. The challenge is that 300 words of text for the pages does not fit into the design appropriately especially on the home page. If I were to use a show/hide element with javascript would this be seen as spam or a trick to the search engines. I do not think it is spam as it will be actual content for the site and the visitor can view it if they click on the show button. Would love to hear your thoughts?
On-Page Optimization | | VivaArturo0 -
Impact of removing category sidebar with keywords?
Our site (a niche financial publication: insideARM.com) requires some more room in the sidebar. We're considering removing the categories (we call them topics) sidebar block, or cutting down the number of items displayed within it. My concern is that we'd be removing a direct link to landing pages for important keyword terms from our most powerful page (the index). Sure, we have the terms listed in the footer, but I am worried that the position change will lower the value of the links. Our users don't really use these links for navigational purposes, which is why it comes up as a potential removed item. Am I wrong to worry about this? Would we be crippling our category pages by doing this?
On-Page Optimization | | insideARM0
