How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best meta description for Category Pages, Tag Pages and Main Article?
Hi, I want to index all my categories and tags. But I fear about duplicating the meta description. for example: I have a tag name "Learn Stock Market", a category name "Learning", and a main article "What is Stock Market". What is your suggestion for meta description of these three pages that looks great for seo google?
On-Page Optimization | | mbmozmb0 -
Https://www.fitness-china.com/hip-thrust-machine title too long
https://www.fitness-china.com/hip-thrust-machine title too long But we have two keywords Hip Thrust Machine and Glute Bridge Machine it 's ok?
On-Page Optimization | | ahislop5740 -
Should I change our main category pages to product listing pages?
With the thought of improving user experience, as well as rankings in Google, I'm considering changing our main category pages to product listing pages (with sub-categories remaining, still). These main category pages are very standard and don't link to any informational content, such as buyers guides, etc. What's driven this is the latest Google core update. I've noticed our main competitor (who we were out-ranking before... but not now) now uses this approach. I can see the benefit from a user perspective, i.e. less clicks to reach products. What's the pros/cons from an SEO point of view, please? Could the potential duplication of content be an issue? For context, we have about 2,000 products and website is on Magento 2.
On-Page Optimization | | alifeofjoy1 -
Blocking Pages E-Commerce Site
Hello, I am working on a site with 1,000's of product pages, some of which do not have inventory in them. Should be blocking these pages in order to reduce bouce rate? How could i manage so many pages efficiently? It would takes weeks to comb through pages to determine which have inventory and which do not. They are also time sensitive as they are live events so dates are always changing. Thanks!
On-Page Optimization | | TP_Marketing0 -
Which is better, a directory 301 redirect or each page in the directory?
A customer of mine has a site with lots of articles and they are all quite spammy. They have not been affected by penguin yet so they asked what to do. I suggested losing the articles directory and 301 redirect to either the home page or another important page. Would a 301 redirect on the entire directory to a single page be the way to go or add redirects from each page within the directory and spread out redirects to various pages in website? Or do you have a better suggestion?
On-Page Optimization | | anthonytjm0 -
Same H1 tag in header across entire site
Should I have the same H1 tag in my header through out my entire site? Or is this considered to be self canalization for my main keywords. For example right now I have an H1 tag with my main targeted keywords on every page on my site, even if the pages content doesn't necessarily match the keywords in the H1 tag.
On-Page Optimization | | TRICORSystems0 -
Robots.txt file
Does it serve any purpose if we omit robots.txt file ? I wonder if spider has to read all the pages, why do we insert robots.txt file ?
On-Page Optimization | | seoug_20050 -
Social media links/buttons - best practices
Has anyone tested social media buttons, to see which types, styles, and placements get the most clicks? Should they go at the top of an article, or are they OK at the bottom? Should the icons definitely have labels? Display # of tweets and Likes? How big should they be? My preference is for discrete buttons with a smallish, plain icon and a label. I don't display tweets or Likes, unless it's a healthy number. And I still think a "share by email" icon is important. I put them at the bottom of the article, to keep the home page uncluttered and lead the eye into reading the article. I'm also concerned about leaking rank from the homepage, especially for a site that's still establishing itself. But if moving buttons to the top gets more shares, that's probably better. Is there a Wordpress plugin that you really like? (I haven't found one yet - I'm still hardcoding my social media buttons.) Opinions are great, but test results are better! Can anyone share?
On-Page Optimization | | mattotoole1