Robots.txt for Facet Results
-
Hi
Does anyone know how to properly add facets URL's to Robots txt?
E.g. of our facets URL -
Everything after the # will need to be blocked on all pages with a facet.
Thank you
-
Great thank you!
-
This is the right answer.
Great way to check is to see if you have multiple versions of that URL indexed, which you don't: https://www.google.com/search?q=site:http://www.key.co.uk/en/key/platform-trolleys-trucks
-
Google ignores everything after the hash to start with, so you do not need to block it to finish with. It is a clever way to pass parameters without having to worry about Google getting lost.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How get google reviews on search results?
Hi, We have good google reviews. (4,8) Can we get this rating stars also on our organic search results ? Best remco
Intermediate & Advanced SEO | | remcoz0 -
If my website do not have a robot.txt file, does it hurt my website ranking?
After a site audit, I find out that my website don't have a robot.txt. Does it hurt my website rankings? One more thing, when I type mywebsite.com/robot.txt, it automatically redirect to the homepage. Please help!
Intermediate & Advanced SEO | | binhlai0 -
Question about Syntax in Robots.txt
So if I want to block any URL from being indexed that contains a particular parameter what is the best way to put this in the robots.txt file? Currently I have-
Intermediate & Advanced SEO | | DRSearchEngOpt
Disallow: /attachment_id Where "attachment_id" is the parameter. Problem is I still see these URL's indexed and this has been in the robots now for over a month. I am wondering if I should just do Disallow: attachment_id or Disallow: attachment_id= but figured I would ask you guys first. Thanks!0 -
Rubber Ball Ranking Results
We noticed a few weeks ago that rankings for the phrase
Intermediate & Advanced SEO | | Jayblue
Charity Collection Buckets
Were bouncing between this page
http://www.carefundraisingsupplies.co.uk/fundraising-products/Charity-Collection-Buckets
Rank 16
to this page
http://www.carefundraisingsupplies.co.uk/fundraising-products/fundraising-supplies
Rank 85
So we de-SEO'd the second page and added more content to the first page.
This seemed to lock Google onto the first page at 16, but it then started to slowly slide downwards. We have made a few more on page text tweaks, tried to reduce keywords density all to no avail. Even though overall this site has a better DA and MOZ profile than those ranked 1 and 2 for the phrase, we just cannot seem to get it moving in the right direction. We are just about to apply some quality links to see if that helps. But we are wondering if we are missing something at a technical level, like category structure, Canonicalisation, 301 redirects or something else. Any thoughts?0 -
Robots.txt vs noindex
I recently started working on a site that has thousands of member pages that are currently robots.txt'd out. Most pages of the site have 1 to 6 links to these member pages, accumulating into what I regard as something of link juice cul-d-sac. The pages themselves have little to no unique content or other relevant search play and for other reasons still want them kept out of search. Wouldn't it be better to "noindex, follow" these pages and remove the robots.txt block from this url type? At least that way Google could crawl these pages and pass the link juice on to still other pages vs flushing it into a black hole. BTW, the site is currently dealing with a hit from Panda 4.0 last month. Thanks! Best... Darcy
Intermediate & Advanced SEO | | 945010 -
Do you add 404 page into robot file or just add no index tag?
Hi, got different opinion on this so i wanted to double check with your comment is. We've got /404.html page and I was wondering if you would add this page to robot text so it wouldn't be indexed or would you just add no index tag? What would be the best approach? Thanks!
Intermediate & Advanced SEO | | Rubix0 -
Odd Results Moving Subdomain Content onto Main Domain
Hi forum! On Thursday night (12/6/12) we moved a page (and all the linking product pages) from our subdomain, mailing-list.consumerbase.com, to our main domain, www.consumerbase.com/mailing-lists.html Shockingly, today I search for "mailing lists" (our #1 target keyword) and we're on the first page! This page never has not ranked well for this keyword in the past. The problem is, the link displaying on Google is our old mailing-list.consumerbase.com subdomain URL. Did moving this content from the new subdomain to our old, well-established domain cause it to appear better in search? Or, since the URL is on the subdomain, did Google just finally get around to indexing that page? Thanks!
Intermediate & Advanced SEO | | Travis-W0 -
Search Engine Blocked by robots.txt for Dynamic URLs
Today, I was checking crawl diagnostics for my website. I found warning for search engine blocked by robots.txt I have added following syntax to robots.txt file for all dynamic URLs. Disallow: /*?osCsid Disallow: /*?q= Disallow: /*?dir= Disallow: /*?p= Disallow: /*?limit= Disallow: /*review-form Dynamic URLs are as follow. http://www.vistastores.com/bar-stools?dir=desc&order=position http://www.vistastores.com/bathroom-lighting?p=2 and many more... So, Why should it shows me warning for this? Does it really matter or any other solution for these kind of dynamic URLs.
Intermediate & Advanced SEO | | CommercePundit0