Robots.txt: excluding URL
-
Hi,
spiders crawl some dynamic urls in my website (example: http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/ + http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/open=true) as different pages, resulting duplicate content of course.
What is syntax for disallow these kind of urls in robots.txt?
Thanks so much
-
You don't want to do this in robots.txt. If you serve pages with these parameters, people will inevitably link to them, and even if they're disallowed in your robots.txt file, Google maybe still index them, according to this: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web."
This is what the rel=canonical tag is designed for. You should use that to tell Google the page is duplicate content of another page on your site, and that it should refer to that other page. You can read (and watch a video) about that here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When using long-tail keywords, should you exactly match for the url or delete "in" "to" etc.?
long-tail keyword - "seizures in adults with no history" Should you include "in and with" in the url?
On-Page Optimization | | Moleculera0 -
Meta Robots index & noindex Both Implemented on Website
I don't want few of the pages of website to get indexed by Google, thus I have implemented meta robots noindex code on those specific pages. Due to some complications I am not able to remove meta robots index from header of every page Now, on specific pages I have both codes 'index & noindex' implemented. Question is: Will Google crawl/index pages which have noindex code along with index code? Thanks!
On-Page Optimization | | Exa0 -
Should I be using the town or city in url with my keyword or keyphrase?
should I be using the town or city in url with my keyword or keyphrase? So lets say I'm trying to rank for butchers in home town should i put the town in the url as well so www.website.com/butchers-in-mytown is that bad? Or would it be best to just put www.website.com/butchers?
On-Page Optimization | | genkee0 -
301 Redirects From a URL without Keyphrases to one With Keyphrases
I have a client that sells services. Each service offered currently has a URL structure like this: www.companyname.com/product/asp$view-id-page3022-item-24 These pages are pretty old, and I would love to have a more user-friendly URL like this: www.companyname.com/product/purple-swatch-watch If I rename the URL and do a 301 redirect, what impact will that have on search? Ideally, this page will be optimized for "purple swatch watch", but the current URL structure is so... well, you know. My apologies if this has been answered before. I tried looking through archived of 301 issues, but lost hope after my first 10 or so attempts at answered didn't help this specific issue.
On-Page Optimization | | ericav0 -
Creating Authority and choosing URL's
Creating Domain Authority and choosing URL's: A: What is better if you want to get higher Domain Authority? Choose keyword.domain.com or www.domain.com/keyword when other sites link to it? B: And for Page Authority? Choose keyword.domain.com or www.domain.com/keyword? Thanks!
On-Page Optimization | | HMK-NL0 -
Adding keywords to URL's
I understand the importance of having the keyword in the URL (at least now I do). When I created my site (www.enchantingquotes.com), I was completely ignorant about SEO. So....question is...how do I go about adding keywords to already done pages? Do I create a new section and then redirect - or do I have to basically recreate pages? Thx much 🙂
On-Page Optimization | | enchantedgirlz0 -
20 x '400' errors in site but URLs work fine in browser...
Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas? example - http://www.morethansport.co.uk/products?sortDirection=descending&sortField=Title&category=women-sports clothing
On-Page Optimization | | Switch_Digital0 -
Keywords in URL:
what kind of URL should we use? www.keyword.net/keyword-city or www.keyword.net/city which URL you would prefer?
On-Page Optimization | | alibeef0