International Sites and Duplicate Content
-
Hello,
I am working on a project where have some doubts regarding the structure of international sites and multi languages.Website is in the fashion industry. I think is a common problem for this industry. Website is translated in 5 languages and sell in 21 countries.
As you can imagine this create a huge number of urls, so much that with ScreamingFrog I cant even complete the crawling.
Perhaps the UK site is visible in all those versions
http://www.MyDomain.com/en/GB/
http://www.MyDomain.com/it/GB/
http://www.MyDomain.com/fr/GB/
http://www.MyDomain.com/de/GB/
http://www.MyDomain.com/es/GB/
Obviously for SEO only the first version is important
One other example, the French site is available in 5 languages and again...
http://www.MyDomain.com/fr/FR/
http://www.MyDomain.com/en/FR/
http://www.MyDomain.com/it/FR/
http://www.MyDomain.com/de/FR/
http://www.MyDomain.com/es/FR/
And so on...this is creating 3 issues mainly:
-
Endless crawling - with crawlers not focusing on most important pages
-
Duplication of content
-
Wrong GEO urls ranking in Google
I have already implemented href lang but didn't noticed any improvements. Therefore my question is
Should I exclude with "robots.txt" and "no index" the non appropriate targeting?
Perhaps for UK leave crawable just English version i.e. http://www.MyDomain.com/en/GB/, for France just the French version http://www.MyDomain.com/fr/FR/ and so on
What I would like to get doing this is to have the crawlers more focused on the important SEO pages, avoid content duplication and wrong urls rankings on local Google
Please comment
-
-
Hey Guido, don't know if it's the best solution, but could be a temporary fix until the best solution is in place. I suggest to move forward with proper HREF LANG tagging or definitely delete those irrelevant languages. Try to do what I said before about validate each country/language and submit a sitemap.xml reflecting that folder to see crawl and index stats pero country/language. Add a sitemap index and obviously validate your entire domain. Just block in the robots.txt unnecessary folders, like images, js libraries, etc. to save crawl budget to your domain.
Let me know if you have another doubt
-
Thank you Antonio, insightful and clear.
There is really not a need of EN versions of localized sites, I think has been done more as was easier to implement (original site is EN-US).
Don't you think robots and noindex EN version of localized sites could be the best solution? for sure is the easier one to implement without affecting UX.
-
Don't know why you have a UK oriented site for German and Italian people, I think is not important those languages in a country mainly English speaking (not US for example, there you must have a Spanish version, or in Canada for English and French). The owner must have their reasons.
Besides this, about your questions:
- If those non-relevant languages must live there, it's correct to implement HREF LANG (may take some time to show results). Also, if the domain is gTLD, you can validate all the subfolders in Google Search Console and choose the proper International targeting. With the ammount of languages and countries I imagine this might be a pain in the ***.
- About the crawling, for large sitesI recommend to crawl per language. If neccesary, per language-country. In this instance I recommend to create a sitemap XML per language or language-country for just HTML pages (hopefully dynamically updated by the e-commerce), create a Sitemap Index in the root of the domain and submit them in Google Search Console (better if you validated the languages or language-country). With this you can answer the question if some language or country are being not indexed with the Submited/Indexed stadistics of GSC.
- Maybe the robots.txt might save your crawl budget, but I'm not a fan of de-index if those folders are truly not relevant (after all, there should be a italian living in UK. If you can't delete the irrelevant langauges for some countries, this can be an option
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
International SEO & Duplicate Content: ccTLD, hreflang, and relcanonical tags
Hi Everyone, I have a client that has two sites (example.com & example.co.uk) each have the same English content, but no hreflang or rel="canonical" tags in place. Would this be interpreted as duplicate content? They haven't changed the copy to speak to specific regions, but have tried targeting the UK with a ccTLD. I've taken a look at some other comparable question on MOZ like this post - > https://moz.com/community/q/international-hreflang-will-this-handle-duplicate-content where one of the answers says **"If no translation is happening within a geo-targeted site, HREFLANG is not necessary." **If hreflang tags are not necessary, then would I need rel="canonical" to avoid duplicate content? Thanks for taking the time to help a fellow SEO out.
International SEO | | ccox10 -
International SEO errors
Hello, In a muddle here. A website has a .co.uk and a .com version. They want to target the UK market and the USA market respectively. The content for the UK version has been localised for the UK audience (e.g. spellings etc) but the content is the same in both sites. There are errors in .co.uk version in webmaster tools : International Targeting | Language > 'en' - no return tags__URLs for your site and alternate URLs in 'en' that do not have return tags.**Q 1) What does this mean?**I can see that both the .com and .co.uk version has only this in place:**2) Should they actually have respectively?**3) Do they also need rel=canonical from the .co.uk to the .comAny help would be appreciated.
International SEO | | AL123al0 -
How to best set up international XML site map?
Hi everyone, I've been searching about a problem, but haven't been able to find an answer. We would like to generate a XML site map for an international web shop. This shop has one domain for Dutch visitors (.nl) and another domain for visitors of other countries (Germany, France, Belgium etc.) (.com). The website on the 2 domains looks the same, has the same template and same pages, but as it is targeted to other countries, the pages are in different languages and the urls are also in different languages (see example below for a category bags). Example Netherlands:
International SEO | | DocdataCommerce
Dutch domain: www.client.nl
Example Dutch bags category page: www.client.nl/tassen Example France:
International domain: www.client.com
Example French bags category page: www.client.com/sacs When a visitor is on the Dutch domain (.nl) which shows the Dutch content, he can switch country to for example France in the country switch and then gets redirected to the other, international .com domain. Also the other way round. Now we want to generate a XML sitemap for these 2 domains. As it is the same site, but on 2 domains, development wants to make 1 sitemap, where we take the Dutch version with Dutch domain as basis and in the alternates we specify the other language versions on the other domain (see example below). <loc>http://www.client.nl/tassen</loc>
<xhtml:link<br>rel="alternate"
hreflang="fr"
href="http://www.client.com/sacs"
/></xhtml:link<br> Is this the best way to do this? Or would we need to make 2 site maps, as it are 2 domains?0 -
Wordpress international SEO Plugin - recommendations needed
Hi Mozzers, I am designing the web architecture for a international website and will be using Wordpress. Can anyone recommend a plugin that lets me SEO for all countries? I have used Yoast many times but it does not seem to work for International Web SEO Architectures. Thanks Carla Here is an idea of what I was thinking of doing Homepage.com Irish-homepage.ie or Homepage.com/ie Irish-Subpages.ie or Homepage.com/ie/subpage Irish-Subpages.ie or Homepage.com/ie/subpage Irish-Subpages.ie or Homepage.com/ie/subpage UK homepage.co.uk or Homepage.com/uk UK-subpages.co.uk or Homepage.com/uk/subpage UK-subpages.co.uk or Homepage.com/uk/subpage
International SEO | | Carla_Dawson0 -
Delivering different content according to country
Hey, I have a question regarding different content according to country (IP)-
International SEO | | Kung_fu_Panda
We planing to serve mobile users using dynamic HTML serving (on the same url)
Is it possible to serve different content for different devices + different IPs (for example different content for a user from US android and someone from UK android ) thanks!0 -
Ranking in Different Countries - Ecommerce site
My client has a .com ecommere site with UK-based serves and he wants to target two other countries (both English speaking). By the looks of it, he wouldn't want to create separate local TLDs targeting each country, I therefore wanted to suggest adding subdomains / subfolders geo-targeted to each country that they want to target, however, I'm worried that this will cause duplicate content issues... What do you think would be the best solution? Any advice would be greatly appreciated! Thank you!
International SEO | | ramarketing0 -
How to optimise a site for 2 countries
Hi there - Any help with the below much appreciated I am helping an Australian company, producing packaging products for businesses. Their site is hosted in Australia and their offices are in Australia. They have asked me to take care of both on-page and off-page SEO so that they rank for keywords related to their products - e.g. 'cardboard boxes'. This should be fairly straightforward for Australian based (.com.au) searchers, but they also supply their products to South Africa, and so want their results to show up also for South African based (.co.za) searchers. Also consider: it is not typical for searchers for these products to use geomodifiers in their search terms there is no unique content for the South African market versus the Australian... the product information is essentially identical. What should we do to ensure their results show up equally for those in South Africa as well as Australia? I am considering building a completely separate site, hosted in South Africa and specifically for the S.A market, but will the duplicate content effect be an issue? Also, this would essentially mean double the SEO effort, is there no way I could achieve our goals more efficiently? many thanks to any help
International SEO | | dnaynay0 -
How do I successfully verify my site for Baidu's webmaster tools?
Instructions for verifying a website via file validation for Baidu's webmaster tools are pretty vague. Does anyone know if the process is the same as Google Webmaster Tools where the verification string must appear in the URL and in the content of the file? Also, does it truly have to be verified within 2.6 hours? Appreciate any feedback from people who have successfully verified their site.
International SEO | | sigmaaldrich0