Welcome to the Q&A Forum

Yup, an "empty" robots.txt file should look like this:

User-Agent: *
 Disallow:

Looks like the site is missing a robots.txt file; that could cause some issues (hit tip, Tim Holmes). I'd get one in place, just for housekeeping - search engines have been known behave oddly sometimes when they can't find a robots.txt file.

I'm struggling to see anything obvious that would cause this behaviour, but equally, I wouldn't be too concerned about it - the page is clearly indexed and discoverable, so this may just be a quirk.

This sounds like the best approach. Typically, most crawlers won't save or interact with cookies, so Google etc will just see the site they've requested.

Hi Sam,

Apologies for the slow response. Your question slipped through the net.

This is an interesting case!

In an ideal world, you'd specify the relationship between all of those pages, in each direction. That's 150+ tags per page, though, which is going to cause some headaches. Even if you shift the tagging to an XML sitemap, that's a _lot _of weight and processing.

Anecdotally, I know that hreflang tagging starts to break at those kinds of scales (even more so on large sites, at that kind of scale, when the resultant XML sitemaps can reach the size of many gigabytes, or when Google is crawling faster than it's processing the hreflang directives), and so tagging everything isn't going to be a viable approach.

I'd suggest picking out and implementing hreflang for _only _the primary combinations*, as you suggest, and reducing the site-wide mapping to the primary variant in each case.

• You might consider that there may be cases where the valuable/primary combinations aren't just the /xx/xx/ or _/yy/yy/ _versions and that there might be some examples of varying country/language combinations which are worth including.

For the atypical variants, I think that you have a few options:

1. Use meta robots (or x-robots) tags to set noindex attributes. This will keep them out of the index, but doesn't guarantee that you're effectively managing/consolidating value across near duplicates - you may be quietly harming performance without realising it, as those pages represent points of crawl and value wastage/leakage.

2. Use robots.txt to prevent Google from accessing the atypical variants. That won't necessarily stop them from showing up in search results, though, and isn't without problems - you risk you creating crawl dead-ends, writing off the value of any inbound links to those pages, and other issues.

3. You use canonical URLs on all of the atypical variations, referencing the nearest primary version, to attempt to consolidate value/relevance etc. However, that risks the wrong language/content showing up in the wrong country, as you're explicitly _un_optimising the location component.

I think that #1 is the best approach, as per your thinking. That removes the requirement to do anything clever or manipulative with hreflang tagging, and fits neatly with the idea that the atypical combinations aren't useful/valuable enough to warrant their own identities - Google should be smart enough to fall back to the nearest 'generic' equivalent.

I'd also take care to set up your Google Search Console country targeting for each country-level folder, to reduce the risk of people ending up in the wrong sections.

Ah, a very interesting question!

I'd not be too concerned; you're loading the content in through a data attribute rather than directly as text. However, there are definitely a few options you could consider:

1. Render via SVG feels like the safest bet, though that's going to be a pretty large, complex set of vectors.
2. Save + serve as an image (and overcome the file size concerns by using WebP, HTTP/2, a CDN like Cloudflare, etc)
3. Serve the content via a dedicated JavaScript file, which you could block access to via robots.txt (a bit fudgey!)

I'd be keen to explore #2 - feels like you should be able to achieve the effect you're after with an image which isn't ridiculously huge.

Yup, that makes sense to me.

It's a bit of a grey area and an unusual case, but I think that this approach makes more sense - otherwise you're actively trying to stop people who aren't in the 'correct' country for a phone number to find/access that page.

Hey Thomas,

Did you have a chance to think about this?

Hmm. This is potentially a little complex.

It sounds like what you're describing _isn't _a case for hreflang / internationalisation.

If I understand correctly, you have one website, which has information about (things in) different regions, but your website isn't explicitly targeting users in different regions?

Where does language come into this?

What happens if I'm a user in Spain, who's Google'd a phone number from Germany? What does the current experience look like (what pages might they see, in what language), and what's the optimal experience look like?

Ah, all very helpful, thanks!

Some interesting bits to pull apart here, I think:

• hreflang tagging is unlikely to improve your rankings for any given language/territory/page/keyword; rather, it's more likely to prevent the wrong content from showing up in the wrong territory. Make sense?

• I'd try to manage expectations around "restoring" rankings. What if your performance dropped because of strong competitor activity, changing consumer behaviour, or other factors? Whilst your international setup is part of that picture, the reality is much more complex. I'd try to shift the conversation away from working out ways to / waiting for your rankings to "restore" from a magic bullet fix, and start talking about the many strategies and tactics you might deploy to improve rankings gradually moving forwards.

• I'd be _really _nervous of outsourced link building. If you're handing money to a third party to get you links, you're only a small amounts of semantics away from buying links outright. What are the doing, exactly? Are you producing exceptionally high quality, useful information and resources, which they're helping to shine a spotlight on - or are you paying them a fee for them to magically acquire links? It feels like, of all the possible risks and causes of your problems, this is the area I'd want to scrutinise the most; and in the meantime, start looking into ways in which your pages can earn links without having to pay a mysterious third party!

Yikes, sorry, I didn't see your responses. Sorry for the delayed response!

Your understanding is correct. It doesn't matter whether the content is duplicated within a single domain, across multiple domains, or multiple subdomains - if it's duplicate but targeting different languages and/or territories, then you need to use hreflang tags to manage that. If the pages are the same in all but phone number (where the phone number is targeting a different territory), that's exactly the kind of situation where you should be using hreflang tags.

Separately, all pages should use canonical tags which reference the correct version of themselves (note, not the alternate/hreflang equivalent), as way of declaring that they're the correct version of that localised page. The hreflang tags take care of the relationship between the multilingual versions without the need to try and do anything clever with pointing canonical references between them.

The blog question is more complex. Are your audiences/products/services different enough in those markets to warrant a distinct content strategy? Does that make sense, strategically?

If the content is different, then you don't want hreflang tags on those individual pages, because you've no equivalent resource/page to reference. Challenging!

So, in a nutshell:

Use hreflang tags everywhere where you've very-similar/duplicate pages which only differ in terms of which territory they serve - regardless of which domain they're on.

Use canonical tags on all pages/templates, to reference the correct version of that page/template - but this shouldn't try and do anything clever in terms of its relationship with the hreflang tags.

Where content isn't the same between multiple sites/pages, don't use hreflang tags (but do still use canonical tags).

Make sense?

Hello!

There are a few parts to this answer; let's pull it apart a little.

• Firstly, setting geographic targeting in search console is unlikely to positively impact your rankings, visibility or traffic from the UK - this tool is more to do with helping Google to understand which users should not find your website, and to help manage brand who have websites with different areas (or multiple websites) which target different countries. That said, there's no harm in enabling it, and I'd recommend that you leave it set.

• It'd also be helpful to understand what you mean when you say that you've "set geo-targeting code at the back end of the website". Are you referring to hreflang tagging? And if so, what does your configuration look like? A partial or erroneous implementation of this can cause more problems than it solves!

  I'd also double-check what your on-page language tags/attributes look like - there are a number of signals which you can send through your language markup, which might potentially help or confuse google.

• I wonder how much of this might be a measurement issue I'd be interested to understand if you've selected the option in Google Analytics to try and filter out common bots and crawlers? It may be that much of the traffic you're seeing from India isn't human.

That rules out most of the technical and measurement challenges. The next areas I'd look are more challenging, and a bit 'bigger picture'.

• Are you using tools like Moz, Search Console and SEMrush and others to measure how and where your website is ranking for various queries? Can you see the kinds of keywords which you're visible for, which are driving this traffic?

• Is your content, brand, product and/or service relevant to a UK audience?

• Does your website provide a good experience for searchers who are looking for the content you provide; and how does that quality of experience compare to other websites who serve that audience (particularly in the UK)?

• Is your website well-constructed, managed, and generally _good _and usefu__l? Is it differentiated and distinct? Is your content well-written and helpful?

• Do other websites, blogs, communities and social audiences link to, talk about, promote and cite your website - again, particularly in the UK?

Things I wouldn't worry about:

• It doesn't really matter where your website is hosted. In an age when most hosting and routing infrastructure is cloud-based and international, this isn't really an issue. Where this _might _affect you is around speed and performance (hosting which is geographically far away from your visitor might mean a slower response) - I'd check with tools like Pingdom and WebPagetest to see how you're performing, and to spot ways to speed things up.

• I'd not worry overly about directories or submission of any kind - any effort you'd spend submitting your site to these kinds of listings could be better spent on improving your content/website/service and engaging with the communities you operate in, with an aim of encouraging people to talk about, cite and engage with your brand.

I appreciate that none of these are easy, quick wins - however, hopefully they'll provide a starting point for you to think about!

Let me know if you've any follow-up thoughts or concerns, or if anything I've signposted leads to any further questions.

Hey Alessia,

So, there are quite a few moving parts here; let's go through them one by one...

When you say that you've changed domain to siteground, what does that mean, exactly? There are a lot of nuances and a lot of potential complexity surrounding changing domains and sites; something I've written about recently on Moz. It'd be good to understand whether this was a just a domain migration (and if so, how it was managed), or something more complex or comprehensive.

If you could answer that, it'd provide a lot of useful context for helping to diagnose your problem, and working towards some options/answers.

In the meantime...

• With regards to the traffic you're still getting from Japan, I wouldn't be too concerned. It's doubtful that it's harming you, although, if it's coming through as a result of a hacked site, I'd wonder if some/all of it might actually be bot traffic - are you tracking it in Google Analytics / similar, and can you get a feel for whether it looks like normal browsing behaviour (or something a little more robotic)?
• With regards to the errors reported in Search Console, I wouldn't worry too much. If you're confident that you've resolved the issue, the error reports are just paperwork - they don't harm you in any way, and you can tick them off a little at a time. Unfortunately, there's no fast way to do this, and you can only remove 100 (I think?) per day, without using their API; but it really doesn't matter too much. Still, it's worthwhile clearing them out, because you'll want to be able to see any new or remaining issues in the future.

Hey Alessia,

Just wanted to ping you a note to let you know that I'm looking at and thinking about this, and intending to get back to you tomorrow with a structured answer and some direction.

Ah, I definitely suspect that our candidate here is the hreflang tagging.

The use-case definitely extends to the same content/language targeting multiple different countries, and it's implemented in exactly the same way. For each page/URL, you'll need to point to the equivalent page on the other site (either via tags or in the XML sitemap).

You'll want each page to have an _en-_au and _en-_zn value.

I'd double-check that you're targeting the right regions for each site in Search Console, too.

I'd be very surprised if that doesn't fix everything pretty much overnight!

Hello!

Could I ask a few questions to help me understand the broader context?

• Are you using hreflang tagging between the sites/content; and if so, are you confident that this is correctly configured and maintained?
• How much is your content localised to the market in question - or is it just copied without variation?
• Does your product/service/price/proposition or availability vary by location?
• How much does the competition vary by territory; who's beating you, and what are they doing differently?
• Has performance dropped on both sites, or just one? And are there particular pages or keywords which have suffered?

If you could probe these areas a little, we might be able to steer towards a better understanding of what's going on.

Hello!

Unfortunately, the answer is very much "it depends"; on the client, their product, their market, and to a degree, how your commercial relationship with them is set up.

Let's explore some key elements, and hopefully, you can factor some of that back into a decision on how you'd like to proceed with them.

• The UK and US are different markets, with different demand, different consumer behaviours, sizes, and competitors. The same keyword in both territories might have a very different competitive landscape, depending on all of these factors.
• Furthermore, your link profile (vs your competitors') likely varies similarly; if you've no local relevance, you're likely going to struggle. You'll be competing against specific local competitors, as well as larger global companies, on multiple fronts.
• Real-world factors will affect you, too. If your product/service is in the UK, how well does it compete against local offerings - particularly in terms of delivery (time, costs), availability (opening hours, response time), etc?
• How well-supported is international targeting on the website? Do you have a different section for each territory (e.g., example.com/us/), and is the website set up with correct hreflang tagging?

In a nutshell, it'll definitely take more work and time to get everything set up, to scale any monitoring and reporting you're doing, and to grow your competitive footprint in both territories.

However, there are economies of scale; you'll have a better understanding of the marketplace overall, and any content or marketing you do could (in some scenarios) benefit both territories. From a billing perspective, I'd expect it to make more sense to run it as a single, large project, rather than as multiple projects.

Hopefully this is helpful; let me know if youve any questions which we haven't explored!

Oh, excellent work!

Good to know for the future!

Hello!

There are three parts to this answer.

Firstly, your technical setup.

Are you confident that the website's SEO and technical configuration is appropriate and correct? For example, are you using hreflang tagging to map out the relationships between your pages, languages and territories - and if so, is it implemented completely and correctly (and, in conjunction with correct canonical tag behaviours), etc?

Are you also using Google Search Console to manage your international targeting?

Secondly, the markets themselves.

As Igor hints at, it's important to understand that these territories, markets and search results are all very different.

Your competitors are likely to vary by market, in terms of their physical presence, their marketing efforts and effectiveness, and their own international SEO + targeting strategies.

More significantly, the users and markets themselves are different - user behaviour, preference, market demands, price sensitivity, and a myriad of other factors can affect how well a brand performs in one locale vs another.

I'd spend some time looking at who's beating you, and to try to understand (from the perspective of a prospective consumer from the market in question) what might be better about their proposition, pricing or other factors.

Thirdly, your linking and promotion strategies.

Is your website naturally acquiring links, citation and social coverage in each of the territories you're competing in? And, what tactics and strategies are you using to grow their authority and footprint? The way in which Google might assess links and these kinds of signals is likely to vary by country, in comparison to what your competitors are doing, and in relation to how narrow or broad your international coverage is.

I'd spend some time assessing - again, from the perspective of a prospective consumer for the regions in question - how well-represented and cited your brand is from reputable sources in those territories.

Sorry that there isn't a simple, easy answer - but there are plenty of places to explore, and I'd be happy to help you probe further!

Hello!

Interesting question. I'd like to probe a little, but let's tackle the easy stuff first...

You can use a canonical URL tag in the header of each of your pages to reference which version you'd like Google to consider the 'correct' version of a page.

For example, on www.domain1.com/page/, you can set a canonical URL tag of www.domain2.com/page/. This acts as a 'strong hint' to Google that you consider these pages to be equivalent, and that you'd like the www.domain2.com version to inherit all of the signals from the www.domain1.com example.

This isn't a perfect solution, mind you. If you still have lots of links (internal or external), equity, coverage or other forms of attention pointing at the www.domain1.com example, this page might still have some of the authority and signals. You're essentially asking Google nicely to move the value, and hoping that they agree that that's OK / the right decision.

From a technical perspective, I'm assuming that your setup will involve serving a single site from both domains, and in which case, the content/tech/templates/URLs are the same, apart from the domain. Assuming that this is the case, you need to make sure that every page is a one-to-one, like-for-like match. You shouldn't point everything at the homepage, for example, and you should also make sure that things like category pages, listings, and other templated or procedurally generated pages also use canonical tagging.

If your tech setup is more complex than this, you'll need to do some thinking on how you 'map' canonical tags between the various versions of your pages and content - something which might require some planning and further investigation.

As an additional consideration, there's no guarantee that the www.domain1.com won't show up in search results if people search for it directly, or if that version of the page has disproportionately high amounts of authority (as I outlined above). And whilst you could use meta robots noindex tags on the www.domain1.com pages _and _canonical tags, there are mixed schools of thoughts on whether this is safe - it may be that Google interprets this as an instruction to inherit the noindex attribute on the www.domain2.com example.

As for your particular scenario, I'd be interested in understanding why you want to maintain the original/current version of the website 'for users'. If I can understand a bit more about the business requirements and what success looks like, it may be that I can refine your options a bit.

I note that some of the other answers have referenced domain forwarding/masking, and 301 redirects. I'd be hesitant to do anything with either, without a better understanding of your setup. Conditional and user-based 301 redirects can be risky if not implemented very carefully (and don't solve for your canonical / equity challenge), and domain forwarding is rarely an SEO-friendly solution (you're just making your website available from more/other domains).

Hopefully this is helpful; it'd be great to dig deeper.

Ah, that's interesting!

We don't get Pandora over here in the UK so it's been a while since I touched it. Does it still require/run in flash? That might go some way to explaining weird referrer data...

Hey Sarah,

Just checking back to see if you're still struggling with this?

Hah!

Ah, now, that's particularly interesting. Got a link to the site handy? Might be worth exploring the implementation.

Yikes, definitely sounds like there's some exploring to be done!

I'm going to see if I can rope in some smart people to help, and get back to you.

Hey there,

My gut feeling is that this sounds like bot traffic.

Many generic systems represent themselves as safari-compatible agents, given that this is one of the main mobile web platforms.

I'd be interested in understanding how this traffic looks when you segment it by other attributes; specifically, screen resolution, colour depth and a few other of the 'system' variables in GA tend to give some good clues as to whether this is human or bot traffic. Have you explored these variables, and seen unusual patterns (such as everybody using the same browser, or the same screen resolution)?

Even if there's variation in these areas, it's worth looking at the patterns over time - there are a lot of systems which, for example, monitor site performance, and as part of this legitimately emulate a variety of browsers... However, they tend to do so consistently, e.g,. at the same time of day.

I'd definitely do some digging to see if you can see patterns which suggest that this isn't real traffic, and then, either way, work out what the next steps are.

I presume that you've enabled the setting in your GA profile to exclude common bots/agents?