Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Value of using spaces or no spaces on product category page varient keywords
Hello, all fellow Mozzers,
On-Page Optimization | | JamesDavison
I have taken over a project and this account, so can't change the username according to MOZ.🙃 We run an eCommerce website, and to me, some of the content is conflicting as some pages have more information content than what I would put in a commerce page, but this is how the boss wants it to work, personally, I would separate the content out.
The page I'm working on:
https://www.longstonetyres.co.uk/tyres/205-70-14.html
and this is an example of the rest of these types of pages, I will be tackling:
https://www.longstonetyres.co.uk/tyres/125-15.html I was tasked to improve SEO ranking, when using the MOZ page grader I had a score of 24 out of 27 83% SEO score and 3-page problems. 7th position in Google for the search term 205/70 R14 As it is a generic product listing page, It was pointless to add to the URL and the Internal links I can't reduce as these are links to products, so I went to reduce the
keyword stuffing and making the page content more natural, this improved the page to 25 out of 27, 87% SEO score and 2-page problems. Improvement to 3rd position in Google, but he wants to chase 1st place to be above his competitors, which is fair enough. It turns out that in the past, they have used this type of page to try and get a high ranking for several search terms, as it is a different variation on a tyre size terms are:
205/70 R14, 205/70R14, 205/70 R 14
205/70 X 14, 205/70X14, 205/70 X14
and so on for all the different ways you can search for this tyre size. He is also convinced Google will see these as different search terms, and while I agree to an extent, this causes Keyword Stuffing on the page, which in turn was harming the rankings. Each product listed on the page already has its own title 205/70 R14, 205/70 HR14 and so on, so my question is. What is the best practice for writing content on these types of pages to gain high rankings for several Keywords, and what value does writing the same keyword with spaces and no spaces have? Any help or advice is welcome, so I have a better understanding of how to approach this for this page and the rest of the site. Cheers Mal0 -
What is the best meta description for Category Pages, Tag Pages and Main Article?
Hi, I want to index all my categories and tags. But I fear about duplicating the meta description. for example: I have a tag name "Learn Stock Market", a category name "Learning", and a main article "What is Stock Market". What is your suggestion for meta description of these three pages that looks great for seo google?
On-Page Optimization | | mbmozmb0 -
URL Structure on Category Pages
Hi, Currently, we having the following URL Structure o our product pages: All Products Pages: www.viatrading.com/wholesale/283/All_Products.html Category Page: www.viatrading.com/wholesale/4/Clothing.html Product Page: www.viatrading.com/wholesale/product/LOAD-HE-WOM/Assorted-High-End-Women-Clothing-Lots.html?cid=4 Since we are going to use another frontend system, we are thinking about re-working on this URL Structure, using something like this: All Products Pages: www.viatrading.com/wholesale-products/ Category Page: www.viatrading.com/wholesale-products/category/ Product Page: www.viatrading.com/wholesale-products/category/product-title/ I understand this is better for SEO and user experience. However, we already have good traffic on the current URL Structure. Should we use same left-side filters on Category Pages as in All Products Page? Since we are using Faceted Navigation, when users filter the Category (e.g. Clothing) they will see same page as Clothing Category Page. Is that an issue for Duplicate Content? Since we are a wholesale company - I understand is using "/wholesale/products/" in URL for all product pages a good idea? If so, should we avoid word "wholesale" in product-title to avoid repeated word in URL? For us, SKU in URL helps the company employees and maybe some clients identify the link. However, what do you think of using the SEO-friendly product-title, and 301 redirect it to www.viatrading.com/BRTA-LN-DISHRACKS/, so 1st link is only used by company members and Canonicalized 2nd is the only one seen by general public? Thank you,
On-Page Optimization | | viatrading10 -
Tags vs. Categories? What should I use?
I'm starting with a blog (self-hosted wordpress) and I'm thinking of the following content structure so that the readers are easily able to locate relevant content: Background: It's a blog which gives people relevant info about government jobs. To start with we will just be publishing information about these jobs but over a period of time also intend to post content that helps readers prepare for these jobs. In other words, right now it's just about detailed job notifications but in the coming months, we shall also post about preparation-related information. Typically, each of the job notifications can be bifurcated like: Jobs basis industry Banking Railways Clinical, etc. Jobs basis company ABC co. DEF co. XYZ co. etc. Jobs basis State / City City 1 City 2, etc. Jobs basis educational qualification Graduation Post-Graduation, etc. Now, I'm seriously confused how should I structure this data from the perspective of Categories & Tags such that it's reader as well as SEO-friendly. Do note that each of the government jobs post ideally falls in a couple of above mentioned categories. Thanks..
On-Page Optimization | | Shalin.TJ0 -
H2s & H3s for Category Navigation
Hi all. I am wondering how best to format a category navigation menu. Currently I don't think we're using H2s correctly on our website. Am I right to think that the top level category e.g. Games should be formatted as an H2 and the sub-categories underneath this should be formatted as H3s (to show a hierarchy)? Is there a limit on how many H2s and H3s you should use? Obviously only one H1 per page. Thanks in advance Paul
On-Page Optimization | | kevinliao0 -
How often should I update category and product content to keep it fresh?
I want to keep our site up to date and fresh with content. How often should I update categories and products pages with content? What angel should I take with categories (new products/services etc.) Thanks Craig
On-Page Optimization | | Towelsrus0 -
Do Parent Categories Hurt SEO?
I have parent categories and subcategories. Will it be harder for the subcategories to rank well because they have a parent category? The URL is longer, for one. I am just wondering if I should not have parent categories. I have one category page doing really well and I am trying to boost the others (most of which are subcategories) and this is a concern for me. Thanks! Edit: I also have a category that has 2 parent categories. I want it automatically in those 2 categories and one of its own. By itself it is very important keyword. Is this ok or should I have it be a parent category?
On-Page Optimization | | 2bloggers0 -
Blog.mysite.com or mysite.com/blog?
Hi, I'm just curious what the majority think of what's the best way to start a blog on your website for SEO benefits. Is it better to have it under a sub domain or a directory? Or does it even matter?
On-Page Optimization | | truckguy770