Robots.txt: excluding URL
-
Hi,
spiders crawl some dynamic urls in my website (example: http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/ + http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/open=true) as different pages, resulting duplicate content of course.
What is syntax for disallow these kind of urls in robots.txt?
Thanks so much
-
You don't want to do this in robots.txt. If you serve pages with these parameters, people will inevitably link to them, and even if they're disallowed in your robots.txt file, Google maybe still index them, according to this: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web."
This is what the rel=canonical tag is designed for. You should use that to tell Google the page is duplicate content of another page on your site, and that it should refer to that other page. You can read (and watch a video) about that here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
From an SEO perspective, which is preferable in the URL for a non-English site: local language or English?
For example, the article is titled ' गर्मी में त्वचा की देखभाल कुछ यूँ करें' (English Meaning: Skin Care Tips for Summer), which one of these three URLs is best for SEO purpose? 1. example.com/ गर्मी-में-त्वचा-की-देखभाल/ (URL in local language, Hindi in this case) 2. example.com/summer-skin-care-tips/ (URL uses English words with English translation) 3. example.com/garmi-mein-twacha-ki-dekhbhaal/ (Hindi words, but written in Roman English) A response will be much appreciated, Amit
On-Page Optimization | | DusBus0 -
URL advice
Hi & thanks for looking, I'm not sure if I've adopted the best SEO URL structure for my site, www.vintageheirloom.com For instance, www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/ Works great for the top level category 'All bags', as I'm trying to keyword authentic designer vintage bags. However the sub categories for instance 'Clutch bags' appears as, www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/vintage-clutch-bags/. As you can see at the moment this URL contains duplicate terms vintage & bags. I'm guessing that duplicate keywords in a url isn't too smart, but should amend with Option 1, 2, 3 or something completely different? Option 1 - keep the top level category url the same, change the subcategory: www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/clutch/ Option 2 - amend the top level category: www.vintageheirloom.com/product-category/authentic-designer/vintage-clutch-bags/ Option 3 - amend the top level category as this: www.vintageheirloom.com/product-category/bags/authentic-designer-vintage-clutch/ By the way I'm using WordPress with Woocommerce. I've asked but it's not possible with some technical issues to remove the /product-category/ section. But each product is for example just: www.vintageheirloom.com/shop/vintage-coach-yellow-duffel-sac-bag/ .... sweet. Thanks again !!
On-Page Optimization | | well-its-1-louder0 -
Importance of URL Structure
We are trying to restructure our onpage SEO and want to make sure we have our URLs correct. The problem is we did the URLs incorrectly in the first place and the ones we currently have are several years olds. We have some URLs such as: http://www.firebrandtraining.co.uk/courses/management/prince2.asp and
On-Page Optimization | | RobertChapman
http://www.firebrandtraining.co.uk/courses/cisco/ccna_2007.asp which are not ideal but user experience aside does it make sense for us to change the URLs and use 301 redirects to the new ones or is the damage done to our natural rankings simply not worth making the change? I have read different articles saying different things, some say that URL structure has little weight (if any weight at all) on rankings while other people seem to say it is quite important. In addition we have heard that changing the URLs with a 301 redirect will cause a large drop in ranking which will take months to recover from and contrarily that 301s are now considered "ok" by Google and we shouldn't see too much change at all in our rankings. Any advice would be much appreciated.0 -
URL Rewrite
(By Google Traductor) Hello, I wanted to ask about some changes that we are evaluating for the issue of passing the url with variables to be more descriptive, for example: http://www.agroads.com.ar/detalle.asp?clasi=139592 tohttp://www.agroads.com.ar/humedimetro-para-cereales-draminski-gmm-139592.html In this case corresponds to the breakdown of a product if you have long published andcan be well positioned to change the title of this position would be lost unless youmanage it with a 301, as one would manage when you have more than 30000 products and title may change several times? There are tools to manage this? Finally, we must apply this to all listed with their respective filters, recommends doingtheir part with 301 redirects and analyze what funciene well to continue with the rest or implement a complete change? I hope I can bring a little light to implement this. Greetings and thanks! Roberto
On-Page Optimization | | romaro0 -
Using magentos own url re-writes
We are changing ecommerce platforms. Is it best to use magentos own url re-writes to redirect every page of a site from its old url to it new one?
On-Page Optimization | | LadyApollo0 -
Long URL in listing job portal
Hello I have job portal and I am listing job offers by: regions, position, sector, language skill For example, when user searchs job in Bratislava, Programmer, information technology, english. My URL is www.presbium.sk/bratislava/pragrammer/information-technology/english Title:Job in Bratislava, Programmer, Information technology, english I know, that URL is too long and no good for SEO. Is it solution? When user chooses max. 2 items for example: www.presbium.sk/bratislava/programmer/ than I put in every next links atribute nofollow and google will index only pages with max. 2 items: www.presbium.sk/bratislava/ or www.presbium.sk/programmer/ or www.presbium.sk/bratislava/programmer/ but not www.presbium.sk/bratislava/protrammer/english/, because english link has atribut nofollow. And I want to ask, what is the best solution for SEO when I am listing job offers by regions, position, sector, language skill and I have than long URL and Title www.presbium.sk/bratislava/pragrammer/information-technology/english Title:Job in Bratislava, Programmer, Information technology, english
On-Page Optimization | | PeterSEO0 -
How to Define Best URL Structure for Product Pages?
I am working on my website to edit structure with help of Google's search engine optimization starter guide. There is really good instruction to define URL structure which help us to perform well over Google's organic search. I have resolved issues regarding category pages but, I have confusion to define best URL structure for product pages. My website's product page URL structure is as follow. http://www.vistastores.com/marketumbrellas-californiaumbrella-slpt758-f13-red.html http://www.vistastores.com/homefurniture-winsomewood-93630.html URL structure is constructed with following terms. 1. Root Category Name (Market Umbrellas or Home Furniture or ....) 2. Brand Name 3. Manufacturer Part Number I am not happy with this structure and also not performing well over Google's organic search. I am thinking to include product name or title tag in URL after root domain. But, it may create very long URL and create issues in organic search display. Does it really matter to perform well over Google's organic search? How can I define best URL structure for product pages?
On-Page Optimization | | CommercePundit0 -
Keywords in URL:
what kind of URL should we use? www.keyword.net/keyword-city or www.keyword.net/city which URL you would prefer?
On-Page Optimization | | alibeef0
