International Sites and Duplicate Content

[guidoampollini]

Hello,

I am working on a project where have some doubts regarding the structure of international sites and multi languages.Website is in the fashion industry. I think is a common problem for this industry. Website is translated in 5 languages and sell in 21 countries.

As you can imagine this create a huge number of urls, so much that with ScreamingFrog I cant even complete the crawling.

Perhaps the UK site is visible in all those versions

http://www.MyDomain.com/en/GB/

http://www.MyDomain.com/it/GB/

http://www.MyDomain.com/fr/GB/

http://www.MyDomain.com/de/GB/

http://www.MyDomain.com/es/GB/

Obviously for SEO only the first version is important

One other example, the French site is available in 5 languages and again...

http://www.MyDomain.com/fr/FR/

http://www.MyDomain.com/en/FR/

http://www.MyDomain.com/it/FR/

http://www.MyDomain.com/de/FR/

http://www.MyDomain.com/es/FR/

And so on...this is creating 3 issues mainly:

1. Endless crawling - with crawlers not focusing on most important pages

2. Duplication of content

3. Wrong GEO urls ranking in Google

I have already implemented href lang but didn't noticed any improvements. Therefore my question is

Should I exclude with "robots.txt" and "no index" the non appropriate targeting?

Perhaps for UK leave crawable just English version i.e. http://www.MyDomain.com/en/GB/, for France just the French version http://www.MyDomain.com/fr/FR/ and so on

What I would like to get doing this is to have the crawlers more focused on the important SEO pages, avoid content duplication and wrong urls rankings on local Google

Please comment

[antonioaraya]

Hey Guido, don't know if it's the best solution, but could be a temporary fix until the best solution is in place. I suggest to move forward with proper HREF LANG tagging or definitely delete those irrelevant languages. Try to do what I said before about validate each country/language and submit a sitemap.xml reflecting that folder to see crawl and index stats pero country/language. Add a sitemap index and obviously validate your entire domain. Just block in the robots.txt unnecessary folders, like images, js libraries, etc. to save crawl budget to your domain.

Let me know if you have another doubt

[guidoampollini]

Thank you Antonio, insightful and clear.

There is really not a need of EN versions of localized sites, I think has been done more as was easier to implement (original site is EN-US).

Don't you think robots and noindex EN version of localized sites could be the best solution? for sure is the easier one to implement without affecting UX.

[antonioaraya]

Don't know why you have a UK oriented site for German and Italian people, I think is not important those languages in a country mainly English speaking (not US for example, there you must have a Spanish version, or in Canada for English and French). The owner must have their reasons.

Besides this, about your questions:

• If those non-relevant languages must live there, it's correct to implement HREF LANG (may take some time to show results). Also, if the domain is gTLD, you can validate all the subfolders in Google Search Console and choose the proper International targeting. With the ammount of languages and countries I imagine this might be a pain in the ***.
• About the crawling, for large sitesI recommend to crawl per language. If neccesary, per language-country. In this instance I recommend to create a sitemap XML per language or language-country for just HTML pages (hopefully dynamically updated by the e-commerce), create a Sitemap Index in the root of the domain and submit them in Google Search Console (better if you validated the languages or language-country). With this you can answer the question if some language or country are being not indexed with the Submited/Indexed stadistics of GSC.
• Maybe the robots.txt might save your crawl budget, but I'm not a fan of de-index if those folders are truly not relevant (after all, there should be a italian living in UK. If you can't delete the irrelevant langauges for some countries, this can be an option