How to exclude URL filter searches in robots.txt
-
When I look through my MOZ reports I can see it's included 'pages' which it shouldn't have included i.e. adding filtering rules such as this one http://www.mydomain.com/brands?color=364&manufacturer=505
How can I exclude all of these filters in the robots.txt? I think it'll be:
Disallow: /*?color=$
Is that the correct syntax with the $ sign in it? Thanks!
-
Unless you're specifically calling out Bing or Baidu... in your Robots.txt file they should follow the same directives as Google so testing with Google's Robots.txt file tester should suffice for all of them.
-
Yes, but what about bing and rest of Search Engine?
-
Adrian,
I agree that there certainly is a right answer to the question posted, as the question asks specifically about one way to manage the issue, being a block of filters in the robots.txt file. What I was getting at is that this may or may not necessarily be the "best" way, and that I'd need to look at your site and your unique situation to figure our which would be the best solution for your needs.
It is very likely that with these parameters a robots.txt file block is the best approach, assuming the parameters aren't added by default into category page or category pagination page navigational links, as then it would affect the bot's ability to crawl the site. Also, if people are linking to those URLs (highly unlikely though) you may consider a robots meta noindex,follow tag instead so the pagerank could flow to other pages.
And I'm not entirely sure the code you provided above will work if the blocked parameter is the first one in the string (e.g. domain.com/category/?color=red) as there is the additional wildcard between the ? and the parameter. I would advise testing this in Google Webmaster Tools first.
- On the Webmaster Tools Home page, click the site you want.
- Under Crawl, click Blocked URLs.
- If it's not already selected, click the Test robots.txt tab.
- Copy the content of your robots.txt file, and paste it into the first box.
- In the URLs box, list the site to test against.
- In the User-agents list, select the user-agents you want (e.g. Googlebot)
-
There certainly is a right answer to my question - I already posted it here earlier today:
Disallow: /*?color=
Disallow: /?*manufacturer=Without the $ at the end which would otherwise denote the end of the URL.
-
Hello Adrian,
The Moz reports are meant to help you uncover issues like this. If you're seeing non-canonical URLs in the Moz report then there is a potential issue for Google, Bing and other search engines as well.
Google does respect wildcards (*) in the robots.txt file, though it can easily be done wrong. There is not right or wrong answer to the issue of using filters or faceted navigation, as each circumstance is going to be different. However, I hope some of these articles will help you identify the best approach for your needs:
(Note: Faceted Navigation is not exactly the same as category filters, but the issues and possible solutions are very similar
)Building Faceted Navigation That Doesn't Suck Faceted Navigation Whiteboard Friday
Duplicate Content: Block, Redirect or Canonical
Guide to eCommerce Facets, Filters and Categories
Rel Canonical How To and Why Not
Moz.com Guide to Duplicate ContentI don't know how your store handles these (e.g. does it add the filter automatically, or only when a user selects a filter?) so I can't give you the answer, but I promise if you read those articles above you will have a very good understanding of all of the options so you can choose which is best for you. That might end up being as simple as blocking the filters in your robots.txt file, or you may opt for rel canonical, noindex meta tag, ajax, Google parameter handling, etc...
Good luck!
-
It's not Google's index that I'm interested in in this case, it's for the MOZ reports. Moz was including over 10,000 'pages' because it was indexing these URLs. Now I know how to edit the robots.txt Moz will be prevented from indexing them again (we only have around 2,000 real pages, not 10,000)
-
I sought out the answer from a developer and got the following reply, so posting here in case it helps someone else:
To exclude pages with color or manufacture in them you can use
Disallow: /*?color=
Disallow: /?*manufacturer=A question mark in your try should be omitted as it denotes the end of the url
-
Hi
I would recommend excluding these in Google Webmaster Tools. Once logged in to your account under the "Crawl" menu you will find "URL Parameters". Find the relevant parameter in the list on this page and you can tell Google not to index these pages.
Hope this helps.
Steve
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do you increase the page footprint, so more links appear on search result?
We want to have more website links to appear in SERP. How to do that?
On-Page Optimization | | WalterHalicki0 -
Canonical URL Category and Tags
Hello, I would like to know that I want to use both category and tags in my blog StylishMahi. If I index both category and tags, should I use canonical URL tag to pass referring to main category. As I want more my categories in SERP results ranking higher? I have also attached a picture. Can someone please confirm? Photo by Moz ZigdWMx
On-Page Optimization | | PratapSingh0 -
Removing old URLs from Google
We rebuilt a site about a year ago on a new platform however Google is still indexing URL's from the old site that we have no control over. We had hoped that time would have 'cleaned' these out but they are still being flagged in HTML improvements in GWT. Is there anything we can do to effect these 'external' dropping out of the indexing given that they are still being picked up after a year.
On-Page Optimization | | Switch_Digital0 -
Do search engines look for keywords in the slug?
Is it OK to split a key phrase into a slug and file name, or should the entire key phrase be in the file name. For example, consider the following articles: How to wash your car.
On-Page Optimization | | ChristopherGlaeser
How to change a tire.How to replace a windshield wiper. Will search engines recognize the "how to" in the following taxonomy: www.domain.com/how-to/wash-your-car/
www.domain.com/how-to/change-a-tire/
www.domain.com/how-to/replace-a-windshield-wiper/ Or, should the "how-to" be included in the file name? Best,Christopher0 -
Redirecting URLS on windows
Could anyone help out here please. A client of ours have reveloped their website from HTML to ASP (helpful!). They have 60 odd pages indexed in Google with the .html extension. We need to do a redirect on these pages so that all link juice is passed to the new pages. What would be the best way to do this please?
On-Page Optimization | | Grumpy_Carl0 -
Image URL's have knocked my sub-pages down (WP)
I had most of my keywords within the top 10 for this site, some were even ranking in the top 5. For a possible minor boost, more-so to cover all the bases, I decided to add images to all of the pages, and they were uploaded as a gallery with most of the image file names being the same as the keyword. Thus, url's were created with our targeted phrases, extending off of the corresponding sub-page. After that, Google quickly picked up the url's to the images and began indexing them, when that occurred the sub-page which was to be the landing page, quickly tanked. Nothing else on-site changed besides the uploading of the images, so I'm sure they're conflicting and for whatever reason Google can't decide which page to index. The page that contains the images used, or the actual intended landing page. With WP I didn't see a way to not have them link to anything at all, and just be static when using a gallery, stock at least. So, my question is how can I quickly alleviate this problem and what should I do in the future to avoid this? I believe if I change link thumbnails to image file instead of attachment page, that should fix the issue... Then, I'll have dead URL's which I suppose I should 301 to the sub-page. Alternatively, is there a better solution that will work, I was also thinking about no-indexing the attachment URL's, but that doesn't seem to be an option.
On-Page Optimization | | JayAdams320 -
SEO and Pagination on search results
Hi Our new site is still in beta mode, and I just wanted to check from an SEO perspective that on search results and on the pagination at the bottom of the page, is there anything we need to do so its compatible with the search engines and not have duplicate content issues etc I have implemented search friendly URL's, but I would welcome any other issues or suggestions you have that will help improve the sites SEO performance The URL of a search results page is http://www.cheapfindergames.com/category-arcade-and-platform
On-Page Optimization | | ocelot0 -
Duplicate product urls
Our site automatically creates shorter urls for the products. There is a rel canonical tag in place, but webmaster tools shows these urls have duplicate title tags. Here is an example: http://www.colemanfurniture.com/holden-desk.htm http://www.colemanfurniture.com/writing-desks-secretary-desks/holden-desk.htm Should the longer url be redirected to the shorter one?
On-Page Optimization | | thappe0