Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Doesn't Google Use My Title Tag and Meta Description?
Hi fellow Moz SEOs, Need your URGENT help! We set an optimised title & meta description for our client websites. These titles are approved by our clients. Before somedays, they checked on Google, noticed the title & meta description were not the same. Next moment, they notified me about this issue. The title & meta description looks fine when I checked the source code. So, why Google use title & meta description differently? For example: Title approved by client: Top Specialist Divorce & Family Lawyer - Yeo & Associates LLC
International SEO | | Verz
Google set our title: Yeo & Associates LLC: Top Specialist Divorce & Family Lawyer Title approved by client: Filing For Divorce Online in Singapore | DivorceBureau®
Google set our title: DivorceBureau®: Filing For Divorce Online in Singapore Title approved by client: Halal Buffet & Bento/Packet Meals Event Caterer Singapore | Foodtalks
Google set our title: Foodtalks - Halal Buffet & Bento/Packet Meals Event Caterer Singapore Title approved by client: Child Care Centre in Singapore| Top Preschool | Carpe Diem
Google set our title: Carpe Diem: Child care Centre in Singapore| Top Preschool Every day, they are requesting me to update Google's title with their approved title. Also, asking me these questions.
Why did this happen?
Why didn't set their recommended title? Is there any way to set our approved titles? Please, help me to find the solution. ASAP Thanks in advance!0 -
Google Analytics Search Console for International Countries
Hi Moz Community, Our e-commerce site is trying to gauge the opportunity of certain queries for specific countries. I'm trying to use the search console data presented in GA to do this. I'm looking at the top queries filtered by each country and also the top landing pages for each country as well. The non filtered data for queries and landing pages is completely different than by country and some if it looks wrong. For instance, our most popular query by impressions shows 0 query impressions in the US once filtered by country. Our site is based in the US so this doesn't make any sense, the same is true for landing pages. Is the queries and landing page data in GA under search console a combination of all countries? Since our target is set to the USA in search console is this data technically US based? How is this data so off? Thanks for answering!
International SEO | | znotes0 -
In the U.S., how can I stop the European version of my site from outranking the U.S. version?
I've got a site with two versions – a U.S. version and a European version. Users are directed to the appropriate version through a landing page that asks where they're located; both sites are on the same domain, except one is .com/us and the other is .com/eu. My issue is that for some keywords, the European version is outranking the U.S. version in Google's U.S. SERPs. Not only that, but when Google displays sitelinks in the U.S. SERPs, it's a combination of pages on the European site and the U.S. site. Does anyone know how I can stop the European site from outranking the U.S. site in the U.S.? Or how I can get Google to only display sitelinks for pages on the U.S. site in the U.S. SERPs? Thanks in advance for any light you can shed on this topic!
International SEO | | matt-145670 -
How to replace my .co.uk site with my .com site in the US Google results
My customer and I are based in the UK. My customer's site, www.blindbolt.co.uk has been around for years. Last year we launched their American site, www.blindboltusa.com. Searching on google.com (tested both via proxy and using the gl=us querystring trick), a search for blind bolt on the US Google returns our www.blindbolt.co.uk site. We would like it to show our www.blindboltusa.com website in US searches. Webmaster tools has the Geographic Target set correctly for each site. Does anyone have any ideas or suggestions please? Thanks.
International SEO | | OffSightIT0 -
What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
International SEO | | Envoke-Marketing2 -
Www.google.rs for SERBIA is mising in Engienes
Why you dont have SERBIA in engines for google - www.google.rs You have Croatia, Bosnia and Hercegovina and Slovenia and Serbia is missing 😞 Best regards kasa
International SEO | | stamparija2 -
Results in Google.co.uk when viewing from US?
If I search Google.co.uk from the United States, will I get the same or different results compared to searching from the UK? If different, what can I use to see the same results as if I am in the UK? web proxy? Thanks for the help!
International SEO | | GSWInbound0 -
Google results inverted in a different country
Hi My site and a rival site are bothl UK based. If I search google.co.uk for a keyword I'm above him, yet if I search the australian results, he's on top. How come? I'd really like to put this right and grind his face into the dirt -- if only as revenge for repeatedly stealing my web content. Thanks
International SEO | | Jeepster0