Disallowing URL Parameters vs. Canonicalizing
-
Hi all,
I have a client that has a unique search setup. So they have Region pages (/state/city). We want these indexed and are using self-referential canonicals.
They also have a search function that emulates the look of the Region pages. When you search for, say, Los Angeles, the URL changes to _/search/los+angeles _and looks exactly like /ca/los-angeles.
These search URLs can also have parameters (/search/los+angeles?age=over-2&time[]=part-time), which we obviously don't want indexed.
Right now my concern is how best to ensure the /search pages don't get indexed and we don't get hit with duplicate content penalties. The options are this:
-
Self-referential canonicals for the Region pages, and disallow everything after the second slash in /search/ (so the main search page is indexed)
-
Self-referential canonicals for the Region pages, and write a rule that automatically canonicalizes all other search pages to /search.
Potential Concern: /search/ URLs are created even with misspellings.
Thanks!
-
-
Just so you know Meta no-index can be applied through the HTML but also through the HTTP header which might make it easier to implement on such a highly generated website
-
Yeah, I know the difference between the two, I've just been in a situation where canonicals were recommended as a means of controlling the preferred page _within an indexation context. _If that makes sense.
My biggest concern is with the creation of URLs from misspellings, which still return search results if it's close enough. The redirects could work. Honestly that wasn't something we considered.
I'm liking the noindex approach. They'd have to write a rule that applies it to every page created with a search parameter, which I think they should be able to do.
If it helps, almost the entire site is run by Javascript. Like...everything.
Thanks for the advice. Much appreciated.
-Brad
-
Robots.txt controls crawling, not indexation. Google will still sometimes index pages they cannot crawl. Canonical tags are for duplicate content consolidation, but are not a hard signal and Google frequently ignores them. Meta no-index tags (or X-robots no-index through the HTTP header, if you cannot apply Meta no-index in the HTML) is a harder signal and is meant to help you control indexation
To be honest if the pages are identical why not just 301 redirect the relevant searches (the top-line ones, which result in pages exactly the same as your regional ones) to the regional URLs? If the pages really are the same, it won't be any different for users except for a small delay during the redirect (which won't really be felt, especially if you are using Nginx redirects)
If you can't do that, you're really left with the Meta no-index tag and the canonical tag. Canonical tags avoid content duplication penalties but are a softer signal and they don't consolidate link equity like 301 redirects do (so in many way, there's not actually that much different between Meta no-index and canonicals, except canonical tags are more complex to set up in the first place as they require a destination field)
I'd probably just Meta no-index all the search URLs. Once Google had swallowed that, I would then (after 2-3 weeks) apply the relevant robots.txt rules
If you apply them both at the same time, Google won't be able to crawl the search URLs (since your robots.txt rule will block them) and therefore they will be blind to your canonical / Meta no index directive(s). So you have to handle de-indexation first, and THEN after that block the crawling to save your crawl allowance a bit
But don't do it all at once or you'll get in an unholy mess!
-
Hi there
Canonical tags prevent problems caused by identical or "duplicate" content across multiple URLs. So in this instance implement the disallow rule on al of the URLs containing /search/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
If a URL canonically points to another link, is that URL indexed?
Hi, I have two URL both talking about keyword phrase 'counting aggregated cells' The first URL has canonical link pointing to the second URL, but if one searches for 'counting aggregated cells' both URLs are shown in the results. The first URL is the pdf, and i need only second URL (the landing page) to be shown in the search results. The canonical links should tell Google which URL to index, i don't understand why both URLs are present in search results? Is 'noindex' for the first URL only solution? I am using Yoast SEO for my website. Thank you for the answers.
Technical SEO | | Chemometec0 -
Why put rel=canonical to the same url ?
Hi all. I've heard that it's good to put the link rel canonical in your header even when there is no other important or prefered version of that url. If you take a look at moz.com and see the code, you'll see that they put the <link rel="<a class="attribute-value">canonical</a>" href="http://moz.com" /> ... pointing at the same url ! But if you go to http://moz.com/products/pricing for example, they have no canonical there ! WHY ? Thanks in advance !
Technical SEO | | Tintanus0 -
SEO friendly url strategy...
Hi guys i just wanted your expert opinion on keywords in urls. The example i'm giving you is in regards to a ecommerce website: Option 1: www.example.com/shop/coffee/coffee-beans/brand-coffee-beans-500gr Option 2: www.example.com/shop/coffee/beans/brand-coffee-beans-500gr We sell coffee so i'll keep the example relevant 🙂 Does it make a difference on how the keywords are stacked throughout? Would the search engine combine the two keywords eg. .../coffee/beans/... or would i be better of having .../coffee/coffee-beans/... and is there a penalty for having the same phrase more than once in the url? I hope my question makes sense... 😉 Looking forward to your opinions and ideas!
Technical SEO | | Immanuel0 -
Www or not www base url
Here is the situation. Developer custom coded a magento commerce shop for a seo client and is having problems adding www to the URL without breaking the site. They wont be able to get this completed until a couple months down the road. We are starting monthly SEO this June. Most directories and websites link to the www version of a site not the non www. What should I expect since we are ranking for the non-www and linking to the www version. In web master tools i'm telling google to display the URL as www.
Technical SEO | | waqid0 -
Keyword and URL
I have a client who has a popular name (like 'Joe Smith'). His blog URL has only his first name and the name of his company in it, like joe.company.com. His blog doesn't rank well at all in the first 3-4 Google SERPs. I was thinking of advising him to change the URL of his blog to joesmith.company.com, and having his webmaster do 301 redirects from the old URL to the new one. Do you think this is a good strategy, or would you recommend something else? I realize ranking isn't just about the URL, it's about links, etc. But I think making his URL more specific to his name could help. Any advice greatly appreciated! Jim
Technical SEO | | JamesAMartin0 -
Compare URLs with 302 redirects
Hello I have a store which was developed in Magento. I have about 8300 errors like this: URL: http://www.theprinterdepo.com/catalog/product_compare/add/product/100/uenc/aHR0cDovL3d3dy50aGVwcmludGVyZGVwby5jb20vcHJpbnRlci1wYXJ0cy5odG1sP3A9NA,,/ 1 Warning 302 (Temporary Redirect) Found 3 days ago <dl> <dt>Redirects to</dt> <dt>http://goo.gl/XMaZg</dt> <dd>Description</dd> <dd>Using a 302 redirect will cause search engine crawlers to treat the redirect as temporary and not pass any link juice (ranking power). We highly recommend that you replace 302 redirects with 301 redirects.</dd> </dl> <a class="more expanded">Minimize</a> These URLs, are generated by magento by the COMPARE feature. In my store we bought an extension called SEO Enterprise Suite and I asked the developers(www.mageworx) about this error. Their answer is: Sorry for the late reply. Our extension adds NOINDEX,FOLLOW tag to compare and cookies pages so that they won't be indexed. I do not think that these redirects can hurt your SEO because these pages won't be indexed at all. The question is: What should I do? Is there anyway that SEOMOZ ignores these URLs? What should I do next, I just dont like to have that HIGH number of errors and warnings. Thank you
Technical SEO | | levalencia10 -
New website with slightly new urls
Hi we recently designed our website in work and changed some of the urls. the old site used to be http://www.example.ie/contact-us.htm now it's is http://example.ie/get-in-touch The problem we are having is with sitelinks (the ones auto generate in the serp) ie: about, contact us, team etc etc. Once cliked on, these OLD links are all going to 404 pages because of the change of url. Help with this would be greatly appreciated - I was thinking of blocking these old sitelinks in google web master.
Technical SEO | | GlenBOB0 -
"/" at the end of a URL
I just noticed that I have the exact same page showing up separately in my Google Analytics reports. One has a "/" at the end and the other does not. Otherwise, these are the exact same URL's. Is this something I need to be aware of from a duplicate content perspective? If so, how do I go about fixing this? I thought the SE's would automatically see that a URL with a "/" at the end is the same as one without, but if that is the case, why is it showing up in my reports as two separate pages?
Technical SEO | | Blockinc0