Disallowing URL Parameters vs. Canonicalizing
-
Hi all,
I have a client that has a unique search setup. So they have Region pages (/state/city). We want these indexed and are using self-referential canonicals.
They also have a search function that emulates the look of the Region pages. When you search for, say, Los Angeles, the URL changes to _/search/los+angeles _and looks exactly like /ca/los-angeles.
These search URLs can also have parameters (/search/los+angeles?age=over-2&time[]=part-time), which we obviously don't want indexed.
Right now my concern is how best to ensure the /search pages don't get indexed and we don't get hit with duplicate content penalties. The options are this:
-
Self-referential canonicals for the Region pages, and disallow everything after the second slash in /search/ (so the main search page is indexed)
-
Self-referential canonicals for the Region pages, and write a rule that automatically canonicalizes all other search pages to /search.
Potential Concern: /search/ URLs are created even with misspellings.
Thanks!
-
-
Just so you know Meta no-index can be applied through the HTML but also through the HTTP header which might make it easier to implement on such a highly generated website
-
Yeah, I know the difference between the two, I've just been in a situation where canonicals were recommended as a means of controlling the preferred page _within an indexation context. _If that makes sense.
My biggest concern is with the creation of URLs from misspellings, which still return search results if it's close enough. The redirects could work. Honestly that wasn't something we considered.
I'm liking the noindex approach. They'd have to write a rule that applies it to every page created with a search parameter, which I think they should be able to do.
If it helps, almost the entire site is run by Javascript. Like...everything.
Thanks for the advice. Much appreciated.
-Brad
-
Robots.txt controls crawling, not indexation. Google will still sometimes index pages they cannot crawl. Canonical tags are for duplicate content consolidation, but are not a hard signal and Google frequently ignores them. Meta no-index tags (or X-robots no-index through the HTTP header, if you cannot apply Meta no-index in the HTML) is a harder signal and is meant to help you control indexation
To be honest if the pages are identical why not just 301 redirect the relevant searches (the top-line ones, which result in pages exactly the same as your regional ones) to the regional URLs? If the pages really are the same, it won't be any different for users except for a small delay during the redirect (which won't really be felt, especially if you are using Nginx redirects)
If you can't do that, you're really left with the Meta no-index tag and the canonical tag. Canonical tags avoid content duplication penalties but are a softer signal and they don't consolidate link equity like 301 redirects do (so in many way, there's not actually that much different between Meta no-index and canonicals, except canonical tags are more complex to set up in the first place as they require a destination field)
I'd probably just Meta no-index all the search URLs. Once Google had swallowed that, I would then (after 2-3 weeks) apply the relevant robots.txt rules
If you apply them both at the same time, Google won't be able to crawl the search URLs (since your robots.txt rule will block them) and therefore they will be blind to your canonical / Meta no index directive(s). So you have to handle de-indexation first, and THEN after that block the crawling to save your crawl allowance a bit
But don't do it all at once or you'll get in an unholy mess!
-
Hi there
Canonical tags prevent problems caused by identical or "duplicate" content across multiple URLs. So in this instance implement the disallow rule on al of the URLs containing /search/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Technical URL SEO question
Hi All, We sell a product on our site which is displayed in cubic metres, from an SEO perspective is it ok to have /3m³ in the URL or should I use 3m3. Thanks All
Technical SEO | | Redooo0 -
Keyword Phrase in URL structure
Wondered the best URL structure, to include a major keyword phrase. Our clients' case is that their domain name is not the main keyword. So should we include the keyword phrase in the URL structure to list all their office locations: A - www.website.com/anxiety-treatment/denver/1001
Technical SEO | | ErnieB
or
B - www.website.com/denver/1001 Would this be considered keyword stuffing? We'd like "A" above to rank for keyword phrases related to "anxiety treatment denver", etc.0 -
Webmaster tools...URL Errors
Hi mozzers, Quick question. Whats the best thing to do about URL errors in webmaster tools. They are all 404s that point from external sites. Many of them are junk spam sites. Should I mark them as "fixed" or just leave them. I'm hoping google is aware it's out of my control if spam sites want to link to 404s on my site. Peter
Technical SEO | | PeterM220 -
Best URL Structure for Product Pages?
I am happy with my URLs and my ecommerce site ranks well over all, but I have a question about product URL's. Specifically when the products have multiple attributes such as "color". I use a header URL in order to present the 'style' of products, www.americanmusical.com/Item--i-GIB-LPCCT-LIST and I allow each 'color' to have it's own URL so people can send or bookmark a specific item. www.americanmusical.com/Item--i-GIB-LPCCT-ANCH1 www.americanmusical.com/Item--i-GIB-LPCCT-WRCH1 I use a rel canonical to show that the header URL is the URL search engines should be indexing and to avoid duplicate content issues from having the exact same info, MP3's, PDF's, Video's accessories, etc on each specific item URL. I also have a 'noindex no follow' on the specific item URL. These header URLs rank well, but when using tools like SEOMoz, which I love, my header pages fail for using rel canonical and 'noindex no follow' I've considered only having the header URL, but I like the idea of shoppers being able to get to the specific product URL. Do I need the no index no follow? Do I even need the rel canonical? Any suggestions?
Technical SEO | | dianeb1520 -
Redirect everything from a certain url
I have a new domain (www.newdomain.com) and and an old domain (www.olddomain.com). Currently both domains are pointing (via dns nameserves) at the new site. I want to 301 everything that comes from the www.oldsite.com to www.newsite.com. I've used this htaccess code RewriteEngine On RewriteCond %{HTTP_HOST} !^www.newsite.com$
Technical SEO | | EclipseLegal
RewriteRule (.*) http://www.newsite.com/$1 [R=301,L] Which works fine and redirects if someone visits www.olddomain.com but I want it to cover everything from the old domain such as www.olddomain.com/archives/article1/ etc. So if any subpages etc are visited from the old domain its redirected to the new domain. Could someone point me in the right direction? Thanks0 -
Will a "blog=example "parameter at the end of my URLs affect google's crawling them?
For example, I'm wondering if www.example.com/blog/blog-post is better than www.example.com/blog/blog-post?blog=example? I'm currently using the www.example.com/blog/blog-post?blog=example structure as our canonical page for content. I'm also wondering, if the parameter doesn't affect crawling, if it would hurt rankings in any way. Thanks!
Technical SEO | | Intridea0 -
Which one is best? Parameters or Meta
I have issue regarding duplicate pages on website as follow. http://www.vistastores.com/review.html?pr_page_id=90344 http://www.vistastores.com/review.html?pr_page_id=90345 I checked my Google webmaster tools and found that Google have already set Parameter with pr_page_id. So, what is this? Will Google index all that pages? Can I use following Meta tag to block indexing? Which one is best?
Technical SEO | | CommercePundit0 -
Root vs. Index.html
Should I redirect index.html to "/" or vice versa? Which is better for duplicate content issues?
Technical SEO | | DavetheExterminator0