Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Filter By Category bad for seo?
Hello Everyone! I know that a single product should not have filter by color option since it will create duplicate content, and you have to use canonical tags to solve it. BUT how about sorting through products via category/brands?
On-Page Optimization | | Safxmed
Filter by category changes the URL of the General shop page (ex: hello.com/Shop/Category1022039 ). This page only displays the products within, no content/ descriptions etc unlike the original category page (ORIGINAL CATEGORY PAGE) Each of these category/brand already have their own individual pages (ex: hello.com/Shop/A). This is the page that will be optimized for content, FAQ, and ranking etc. Unlike in the url created when filtering through the categories. So technically I would have 2 URL for each Brand/Category. Would they compete with each other? What would you guys suggest. Please advise me on this. Thank You0 -
Does DA/PA have any effect on rankings?
I have seen many people are concerned about increasing DA and PA of their websites. While I am very curious why do people focus on increasing DA and PA? Does DA and PA effect the rankings of the website? Because I have recently launched my website regarding men beard trimmer and it is ranking on 1st page but not on number 1 position. Will increasing DA/PA of the site help me in occupying 1st position?
On-Page Optimization | | RyanAmin0 -
Category pages, should I noindex them?
Hi there, I have a question about my blog that I hope you guys can answer. Should I no index the category and tag pages of my blog? I understand they are considered as duplicate content, but what if I try to work the keyword of that category? What would you do? I am looking forward to reading your answers 🙂
On-Page Optimization | | lucywrites0 -
Meta Robots index & noindex Both Implemented on Website
I don't want few of the pages of website to get indexed by Google, thus I have implemented meta robots noindex code on those specific pages. Due to some complications I am not able to remove meta robots index from header of every page Now, on specific pages I have both codes 'index & noindex' implemented. Question is: Will Google crawl/index pages which have noindex code along with index code? Thanks!
On-Page Optimization | | Exa0 -
Do Parent Categories Hurt SEO?
I have parent categories and subcategories. Will it be harder for the subcategories to rank well because they have a parent category? The URL is longer, for one. I am just wondering if I should not have parent categories. I have one category page doing really well and I am trying to boost the others (most of which are subcategories) and this is a concern for me. Thanks! Edit: I also have a category that has 2 parent categories. I want it automatically in those 2 categories and one of its own. By itself it is very important keyword. Is this ok or should I have it be a parent category?
On-Page Optimization | | 2bloggers0 -
What is the right schema.org link for a web design / developer / mobile agency?
It seems strange that a group of web developers would make up an entire structured language to designate businesses by category and somehow forget to include companies like.... web developers. So I must be missing it, what is correct to use?
On-Page Optimization | | yeagerd0 -
Does Google index dynamically generated content/headers, etc.?
To avoid dupe content, we are moving away from a model where we have 30,000 pages, each with a separate URL that looks like /prices/<product-name>/<city><state>, often with dupe content because the product overlaps from city to city, and it's hard to keep 30,000 pages unique, where sometimes the only distinction is the price & the city/state.</state></city></product-name> We are moving to a model with around 300 unique pages, where some of the info that used to be in the url will move to the page itself (headers, etc.) to cut down on dupe content on those unique 300 pages. My question is this. If we have 300 unique-content pages with unique URL's, and we then put some dynamic info (year, city, state) into the page itself, will Google index this dynamic content? The question behind this one is, how do we continue to rank for searches for that product in the city-state being searched without having that info in the URL? Any best practices we should know about?
On-Page Optimization | | editabletext0 -
Impact of removing category sidebar with keywords?
Our site (a niche financial publication: insideARM.com) requires some more room in the sidebar. We're considering removing the categories (we call them topics) sidebar block, or cutting down the number of items displayed within it. My concern is that we'd be removing a direct link to landing pages for important keyword terms from our most powerful page (the index). Sure, we have the terms listed in the footer, but I am worried that the position change will lower the value of the links. Our users don't really use these links for navigational purposes, which is why it comes up as a potential removed item. Am I wrong to worry about this? Would we be crippling our category pages by doing this?
On-Page Optimization | | insideARM0