Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Google Is Changing our Title Tags?
Hi fellow Moz SEOs, Need your URGENT help! We set an optimised title for our client websites. These titles are approved by our clients. When they checked on Google, noticed the title was not the same. They notified me about this issue. The title looks fine when I checked the source code. Why Google set our title differently? For example: Title approved by client: Heart Specialist Clinic Singapore | Cardiology Clinic | Dr. Lim Ing Haan
International SEO | | Verz
**Google set our title: **Dr. Lim Ing Haan: Heart Specialist Clinic Singapore ... Title approved by client: Hernia Surgery Singapore | Arden JR Surgery
**Google set our title: **Arden JR Surgery: Hernia Surgery Singapore Title approved by client: Top Specialist Divorce & Family Lawyer - Yeo & Associates LLC
Google set our title: Yeo & Associates LLC: Top Specialist Divorce & Family Lawyer Title approved by client: Child care Centre in Singapore| Top Preschool | Carpe Diem
Google set our title: Carpe Diem: Child care Centre in Singapore| Top Preschool Every day, they are requesting me to update Google's title with their approved title. Also, asking me these questions.
Why did this happen?
Why didn't set their recommended title? Is there any way to set our approved titles? Please, help me to find the solution. ASAP Thanks in advance!0 -
International SEO Two Subdomains Showing Up in Google Search Results
Hi I have a client that is having two subdomains showing up SERP when you Google their name. Here are the details. They have two subdomains us.companyname.com and en.companyname.com us.companyname.com is for the US and has completely different products and content than en.companyname.com en.companyname.com is the site designed for Europe and it is in English. How can I make it so that only the us. version shows up in the search results? Thanks in advance!
International SEO | | JohnWeb120 -
What's the Best Strategy for Multiregional Targeting for Single Language?
I have a service based client who is based in the US but wants to expand to audiences in Australia, Canada, and the United Kingdom. Currently, all the content is in American English with international targeting in Google Search Console set to the US. I know that is going to have to change, but I'm unsure of the best strategy. Right now there are a few basic strategies in my head. Remove International Targeting in GSC and let her rip Remove International Targeting in GSC, install copies of the site on subfolders /au/, /ca/, and /uk/, add hreflang tags, and add canonicals pointing back to original Remove International Targeting in GSC, install copies of the site on subfolders /au/, /ca/, and /uk/, add hreflang tags, and risk duplicate content Have independent writers overcharge for English translations into different dialects and add hreflang tags It's hard to come up with a perfect solution for content differentiation by region in order to implement hreflang tags with a region (en-au, en-ca, en-gb). Remove International Targeting in GSC and let her rip This one is pretty simple. However, I am completely unsure of its effectiveness. Remove International Targeting in GSC, install copies of the site on subfolders /au/, /ca/, and /uk/, add hreflang tags, and add canonicals pointing back to original The point of adding canonicals is to avoid the duplicate content, but then my new subfolders do not get indexed. I'm unsure of what type of exposure these URLs would receive or how they would be valuable. Remove International Targeting in GSC, install copies of the site on subfolders /au/, /ca/, and /uk/, add hreflang tags, and risk duplicate content High risk of a penalty with duplicate content, but my targeting will be the most efficient. Have independent writers overcharge for English translations into different dialects and add hreflang tags This is probably the safest bet, takes the longest, and costs the most money. However, how different will the content actually be if I change truck to lorry, trunk to boot, and optimization to optimisation? Maybe I'm missing something, but this conundrum seems extremely difficult. Weighing the cost, time, and possible result is challenging. Hit me with your best answer and thanks for taking a look at someone else's problem.
International SEO | | ccox12 -
Geo-Targeting separate TLD's where both are .com domains
Hi I have a client who owns two separate TLDs for the same brand (for the sake of this post, we'll call the two sites www.site-a.com and www.site-b.com). For site www.site-a.com the website has been around for a while and is their primary site for their US operations which is their heartland, is well established in the SERPS and is where they make most of their money. As they looked to expand to the UK, they then created www.site-b.com and added the UK as a subfolder (so www.site-b.com/uk) and geo-targeted it towards the UK in Webmaster tools . The site has recently launched but they now find that, when a customer searches for their brand in the UK, they find www.site-a.com in position 1 (which, given it's tailored for a primary US audience, has a significantly lower conversion rate for UK traffic) and www.site-b.com in position 2. However, the client doesn't want to specifically geo target www.site-a.com to the USA as they feel it might affect where they appear for other international markets aside from the UK. So the question is, how can they, with the existing infrastructure, help remove www.site-a.com from the UK SERPs without adversely affecting their rank elsewhere? Hope this makes sense and thanks in advance for your help. James
International SEO | | jimmygs19820 -
Geographic Target set up in Google webmaster tool
Hi, When i launched my web site 3 months ago ( I'm am very new to SEO) I have set up the geographical target section in Google webmaster tool for US. Now, I'm thinking to change it to some other geo target to see if i can get more traffic. However, recently few of my prompted keywords got really well in Google US ranking. Here are my Questions: if i will change the geo settings in webmaster tool will effect the ranking i already managed to achieve in US? In the list of all the countries in Google webmaster tools what does is mean "unlisted"? Can i select more than one country to target and if I can how? Thanks!! Raviv
International SEO | | Indiatravelz0 -
Analytics Profile for '.co.uk' extension
What's up Mozzers, I am currently doing some work for a local business in the UK and they've asked me to set-up Google Analytics and Webmaster Tools profiles for them. The site is currently accessible at the following domain extension, 'mydomain.com' and 'mydomain.co.uk'. What is the best way to set this up in Google Analytics and Google Webmaster Tools so I get the most accurate measurement of traffic coming to my website?
International SEO | | NiallSmith0 -
Google UK picking up USA Site
I have a site with two subfolders one is .../uk and one is .../us Part of the content on the two sites is the same and part is unique. The US site's language is set to en and the UK site's language is set to en_gb. I have setup geo-targeting in webmaster tools. The problem is that the home page is a GEO-IP redirect and it seems to be picking up information from the US site even on google uk. I'm not concerned too much about getting the uk site crawled as we submit a sitemap for that anyway. But my concern is that if I setup the geo-ip redirect as a 301 will my UK site loose all of it's ranking? Also am I likely to be penalised for duplicate content?
International SEO | | matthewdolman0 -
Results in Google.co.uk when viewing from US?
If I search Google.co.uk from the United States, will I get the same or different results compared to searching from the UK? If different, what can I use to see the same results as if I am in the UK? web proxy? Thanks for the help!
International SEO | | GSWInbound0