Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I outrank a website whose brand is named specifically after a product / service (and mine is not!) ?
Hi everyone, I've been working hard on my on-page SEO lately so I can gain visibility for my website. The results have been great and I am now on top of the SERP for the services I propose. I think that my content is almost fully optimize and** I've respected all the best practices (both on-page and technical SEO)**. However, there is one problem I just can't deal with for I don't have the knowledge, hence this post. I'm pretty sure that this issue is quite common for SEO experts. Here the thing: I offer dental emergency services, which is my core business. I'm ranked 4th for that request in my area, which is good, but I'd like to rank 1st as** I have a better DA and content than the 3 websites outranking me**. Also, I'm the first result for any other related services such as "dental services" or "dentist". However, when it comes to the theme "dental emergency", I'm constantly outranked by the same 3 websites. I ran an audit on their website but **my content and technical SEO is way better **than theirs. I suppose that the only reason I'm behind them is because they used "dental emergency" in their **Brand name **and, therefore, in the Home page URL. Every time someone is looking up online for "dental emergency", these websites will be on top of the SERP as I think that Google is unable to know whether the users are specifically looking for their websites (aka Brand) or for "dental emergency" services. Here is an example of a competitor: https://www.urgencedentairedemontreal.com/ (urgence meaning emergency in French). His whole Brand name and URL have been built after the "dental emergency." service. On the contrary, **my Brand name does not mention "urgence". ** I see that as a trick that is confusing Google. The fact that my competitors named their Brand after a specific service I also offer is real pain for my SEO. I also think it's really unfair as I've put a lot of effort in designing a nice website with great UX and content. This is the kind of practice that should be penalized in my opinion. Please, does anyone know any way to resolve this issue?
On-Page Optimization | | AlexTL0 -
Using h2 for category on ecommerce website
Hi, I am working on an ecommerce site and the main category - lets call them car widgets - is using a h1 at the top of the page which is great. There are 4 sub categories on the page - lets call one of them red widget. The only content on the page is the sub category name and an image. Should the sub category red widget use a h2? Thanks S
On-Page Optimization | | bedynamic0 -
Using keywords in my URL: Doing a redirect to /keyword
My website in "On Page Grade" received an A.Anyway, I only have 1 thing to optimize:_"Use Keywords in your URL__Using your targeted keywords in the URL string adds relevancy to your page for search engine rankings, assists potential visitors identify the topic of your page from the URL, and provides SEO value when used as the anchor text of referring links."_My website is ranking in top10 for a super high competitive keyword and all my others competitors have the keyword on their domain, but not for my URL.Since I can't change my domain for fixing this suggestion, I would like to know what do you think about doing a 301 redirect from / to mydomainname.com/keyword/So the index of my website would be the /keyword.I don't know if this can make a damage to my SERP for the big change ir it would be a great choice.
On-Page Optimization | | estebanseo0 -
How to Handle duplicate pages/titles in Wordpress
The wordpress blog causes problems with page titles. If you go to the second page of blog posts it there's a different URL but with the same page title. for example: page 1: site/blog page 2: site/blog/page/2 Each page gets flagged for duplicate page titles. Thanks in advance for your thoughts,
On-Page Optimization | | heymarshall1 -
Best practice for footer in ecommerce - Shall I add Top Category links?
What would you recommend regarding links to "Top Products" and "Top Categories" in footer? Would you add them to give extra link juice to top categories? would you try to avoid category links in footer that are already in the header navigationor in the main content area to avoid linking twice from all pages? would you vary these top category links in footer according to main category
On-Page Optimization | | lcourse0 -
Best SEO Extension/Plugin for NOPCommerce Site?
Hi I am working for a client who is using NOPCommerce. It doesn't look like they have a SEO Plugin in - although you can add meta descriptions to Products - which works fine, the Product categories have SEO components too but do not seem to work and all 'other' content /CMS pages have no SEO components whatsoever. Does anyone know of a plugin which would resolve this? (PS never used NOPCommerce before!)
On-Page Optimization | | AllieMc0 -
How to properly remove pages and a category from Google's index
I want to remove this category http://www.webdesign.org/web-design-news-all/ and all the pages in that category (e.g. http://www.webdesign.org/web-design-news-all/7386.html ) from Google's index. I used the following string in the "Reomval URS" section in Google Webmaster Tools: http://www.webdesign.org/web-design-news-all/* is that correct or I better use http://www.webdesign.org/web-design-news-all/ ? Thanks in advance.
On-Page Optimization | | VinceWicks0 -
Same H1 tag in header across entire site
Should I have the same H1 tag in my header through out my entire site? Or is this considered to be self canalization for my main keywords. For example right now I have an H1 tag with my main targeted keywords on every page on my site, even if the pages content doesn't necessarily match the keywords in the H1 tag.
On-Page Optimization | | TRICORSystems0