International Sites and Duplicate Content
-
Hello,
I am working on a project where have some doubts regarding the structure of international sites and multi languages.Website is in the fashion industry. I think is a common problem for this industry. Website is translated in 5 languages and sell in 21 countries.
As you can imagine this create a huge number of urls, so much that with ScreamingFrog I cant even complete the crawling.
Perhaps the UK site is visible in all those versions
http://www.MyDomain.com/en/GB/
http://www.MyDomain.com/it/GB/
http://www.MyDomain.com/fr/GB/
http://www.MyDomain.com/de/GB/
http://www.MyDomain.com/es/GB/
Obviously for SEO only the first version is important
One other example, the French site is available in 5 languages and again...
http://www.MyDomain.com/fr/FR/
http://www.MyDomain.com/en/FR/
http://www.MyDomain.com/it/FR/
http://www.MyDomain.com/de/FR/
http://www.MyDomain.com/es/FR/
And so on...this is creating 3 issues mainly:
-
Endless crawling - with crawlers not focusing on most important pages
-
Duplication of content
-
Wrong GEO urls ranking in Google
I have already implemented href lang but didn't noticed any improvements. Therefore my question is
Should I exclude with "robots.txt" and "no index" the non appropriate targeting?
Perhaps for UK leave crawable just English version i.e. http://www.MyDomain.com/en/GB/, for France just the French version http://www.MyDomain.com/fr/FR/ and so on
What I would like to get doing this is to have the crawlers more focused on the important SEO pages, avoid content duplication and wrong urls rankings on local Google
Please comment
-
-
Hey Guido, don't know if it's the best solution, but could be a temporary fix until the best solution is in place. I suggest to move forward with proper HREF LANG tagging or definitely delete those irrelevant languages. Try to do what I said before about validate each country/language and submit a sitemap.xml reflecting that folder to see crawl and index stats pero country/language. Add a sitemap index and obviously validate your entire domain. Just block in the robots.txt unnecessary folders, like images, js libraries, etc. to save crawl budget to your domain.
Let me know if you have another doubt
-
Thank you Antonio, insightful and clear.
There is really not a need of EN versions of localized sites, I think has been done more as was easier to implement (original site is EN-US).
Don't you think robots and noindex EN version of localized sites could be the best solution? for sure is the easier one to implement without affecting UX.
-
Don't know why you have a UK oriented site for German and Italian people, I think is not important those languages in a country mainly English speaking (not US for example, there you must have a Spanish version, or in Canada for English and French). The owner must have their reasons.
Besides this, about your questions:
- If those non-relevant languages must live there, it's correct to implement HREF LANG (may take some time to show results). Also, if the domain is gTLD, you can validate all the subfolders in Google Search Console and choose the proper International targeting. With the ammount of languages and countries I imagine this might be a pain in the ***.
- About the crawling, for large sitesI recommend to crawl per language. If neccesary, per language-country. In this instance I recommend to create a sitemap XML per language or language-country for just HTML pages (hopefully dynamically updated by the e-commerce), create a Sitemap Index in the root of the domain and submit them in Google Search Console (better if you validated the languages or language-country). With this you can answer the question if some language or country are being not indexed with the Submited/Indexed stadistics of GSC.
- Maybe the robots.txt might save your crawl budget, but I'm not a fan of de-index if those folders are truly not relevant (after all, there should be a italian living in UK. If you can't delete the irrelevant langauges for some countries, this can be an option
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
International SEO - Targeting US and UK markets
Hi folks, i have a client who is based in italy and they set up a site that sells travel experiences in the sout of Italy (the site currently sit on a server in Italy). The site has been set up as gTLDs: www.example.com They only want to target the US and the UK market to promote their travel experiences and the site has only the english version (the site does not currently offer an italian version). If they decide to go for the gTLDs and not actually change to a ccTLDs (which would be ideal from my point of view) how are the steps to be taken to set this up correctly on GSC? They currently only have one property registered on GSC: www.exapmple.com therefore i guess the next steps are: Add new property - www.example.com/uk and and set up geo targeting for UK Existing property - www.example.com/ set up geo targeting for US In case the client does not have the budget to optimise the content for american and british languages, would still make sense to have 2 separate property in GSC (example.com for US market and example.com/uk for UK market)? Few considerations: Add canonical tag to avoid duplicate content across the two versions of the site (in the event there is no budget to optimise the content for US and UK market)? Thank you all in advance for looking into this David
International SEO | | Davide19840 -
How to avoid duplication across multiple country domains
Here's the scenario: I have a client currently running one Shopify site (AU) They want to launch three more country domains (US, UK and EU) They want each to be a standalone site, primarily so the customers can purchase in their local currency, which is not possible from a single Shopify site The inventory is all from the same source The product desscriptions will all be the same as well Question: How do we avoid content duplication (ie. how will canonical tags work in this scenario)?
International SEO | | muzzmoz0 -
Local Strategies For International Business
A client of mine is in a field that Google (correctly) recognizes as international, and does get traffic and leads for this but they are bound by sales covenants to sell only in regional geographic territories in their country, Other than PPC is there a strategy I can use to increase regional traffic? As mentioned, Google does not recognize this business as local.
International SEO | | waynekolenchuk0 -
How to handle rel canonical on secondary TLD's - multi regional sites.
I currently have a .com domain which I am think of duplicating the content on to another tld, CO.UK (and regionalize some aspects like contact numbers etc). From my research, it seems that in gwt you must then indicate which country you wish to target, in the co.uk case the UK. My question is how should I handle rel canonical in the duplicated site. should it rel canonical back to the .com or the co.uk? Any other pointers would also be appreciated. Thx
International SEO | | dmccarthy0 -
Site with multiple languages
We are building a Joomla site for a customer that has an USA division and a South American division (english and spanish). The products and services are the same. I am trying to understand the best posible way to architect the site. 1- Do I create 1 site with duplicate pages in different languages? Does Google recognize that it is duplicate content if different languages are used? 2- Do I create seperate sub domains for each language? 3- Should I just use Google translate to translate the pages as required? The problem here is that each site has a different geographic target. any other alternatives?
International SEO | | brantwadz0 -
How to optimise you site in other countries eg Australia
We would like to rank better for specific keywords in Australia. We rank pretty well in our home tld .co.uk but would like to do so in .com.au I would appreciate your thoughts and recommendations.
International SEO | | seanmccauley0 -
Multi-lingual SEO: Country-specific TLD's, or migration to a huge .com site?
Dear SEOmoz team, I’m an in-house SEO looking after a number of sites in a competitive vertical. Right now we have our core example.com site translated into over thirty different languages, with each one sitting on its own country-specific TLD (so example.de, example.jp, example.es, example.co.kr etc…). Though we’re using a template system so that changes to the .com domain propagate across all languages, over the years things have become more complex in quite a few areas. For example, the level of analytics script hacks and filters we have created in order to channel users through to each language profile is now bordering on the epic. For a number of reasons we’ve recently been discussing the cost/benefit of migrating all of these languages into the single example.com domain. On first look this would appear to simplify things greatly; however I’m nervous about what effect this would have on our organic SE traffic. All these separate sites have cumulatively received years of on/off-site work, and even if we went through the process of setting up page-for-page redirects to their new home on example.com, I would hate to lose all this hard-work (and business) if we saw our rankings tank as a result of the move. So I guess the question is, for an international business such as ours, which is the optimal site structure in the eyes of the search engines; Local sites on local TLD’s, or one mammoth site with language identifiers in the URL path (or subdomains)? Is Google still so reliant on TLD for geo targeting search results, or is it less of a factor in today’s search engine environment? Cheers!
International SEO | | linklater0 -
Internationally targetted subdomains and Duplicate content
A client has a site they'd like to translated into French, not for the french market but for french speaking countries. My research tells me the best way to implement this for this particular client is to create subfolders for each country. For ease of implementation I’ve decided against ccTLD’s and Sub Domains. So for example… I'll create www.website.com/mr/ for Mauritania and in GWT set this to target Mauritania. Excellent so far. But then I need to build another sub folder for Morocco. I'll then create www.website.com/ma/ for Morocco and in GWT set this to target Morocco. Now the content on these two sub folders will be exactly the same and I’m thinking about doing this for all French speaking African countries. It would be nice to use www.website.com/fr/ but in GWT you can only set one Target country. Duplicate content issues arise and my fear of perturbing the almighty Google becomes a possibility. My research indicates that I should simply canonical back to the page I want indexed. But I want them both to be indexed surely!? I therefore decided to share my situation with my fellow SEO’s to see if I’m being stupid or missing something simple both a distinct possibility!
International SEO | | eazytiger0