Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Doesn't Google Use My Title Tag and Meta Description?
Hi fellow Moz SEOs, Need your URGENT help! We set an optimised title & meta description for our client websites. These titles are approved by our clients. Before somedays, they checked on Google, noticed the title & meta description were not the same. Next moment, they notified me about this issue. The title & meta description looks fine when I checked the source code. So, why Google use title & meta description differently? For example: Title approved by client: Top Specialist Divorce & Family Lawyer - Yeo & Associates LLC
International SEO | | Verz
Google set our title: Yeo & Associates LLC: Top Specialist Divorce & Family Lawyer Title approved by client: Filing For Divorce Online in Singapore | DivorceBureau®
Google set our title: DivorceBureau®: Filing For Divorce Online in Singapore Title approved by client: Halal Buffet & Bento/Packet Meals Event Caterer Singapore | Foodtalks
Google set our title: Foodtalks - Halal Buffet & Bento/Packet Meals Event Caterer Singapore Title approved by client: Child Care Centre in Singapore| Top Preschool | Carpe Diem
Google set our title: Carpe Diem: Child care Centre in Singapore| Top Preschool Every day, they are requesting me to update Google's title with their approved title. Also, asking me these questions.
Why did this happen?
Why didn't set their recommended title? Is there any way to set our approved titles? Please, help me to find the solution. ASAP Thanks in advance!0 -
Getting accurate Geo Location traffic stats in Google Analytics - HELP
One of our clients services the US and the UK, but having looked at the report over an extended period of time we can still see that the vast majority of traffic is coming from the US. I.e. our last report for March indicated that there were over 3,000 users in the US but only 6 in the UK. We know that Google Analytics works out a user’s location based on where their IP is located and not their physical location, and that this means that the data needs to be taken with a pinch of salt as it won’t always represent what you expect. That being said, we know that the traffic figures for Europe are largely inaccurate and would like to get some more accurate stats to report on. Is there a way to do so at all within Google Analytics?
International SEO | | Wagada1 -
Will hreflang with a language and region allow Google to show the page to all users of that language regardless of region?
I'm launching translations on a website with the first translation being Brazilian Portuguese. If I use the following hreflang: If a user is outside of Brazil and has their browser language set to just Portuguese (Not Portuguese (Brazil)) will Google still serve them the Portuguese version of my pages in search results?
International SEO | | Brando160 -
What's the best homepage experince for an international site?
Greeting Mozzers. I have a question for the community, which I would appreciate your input on. If you have a single gTLD that services multiple countires, what do you think is the best homepage UX for the root homepage and why? So the example would be you own website www.company.org and target content to Germany, Japan and Australia with content through the folder structure eg. www.company.org/de-de If someone comes to the www.company.org from a region, would you: Redirect them based on location IP – so if from Germany they land on www.company.org/de-de Let them land on the homepage which offers location selection Let them land on a page with content and offer location selection eg. pop-up or obvious selection box Something I’ve not thought of… I'd appreciate your input. Thanks
International SEO | | RobertChapman0 -
Other country TLD's for US product
We have a product ( Example: Car ) where all of the TLD's for North America (Example: Car.com, Car.net, etc) have been taken. I've found several for TLD's like .IT, .LA, .AG, etc. If I purchased those and launched sites under those TLD's in the US on servers here in the US and marketed the same as a North American TLD, do you see any issues with this regarding SEO challenges? Thanks All! Hugs, Natalie 🙂
International SEO | | okiedokie0 -
Country Specific Google Results
Does anyone have any stats (preferred) on users selecting Google results segmented to their country? For instance, users in the UK (France, Japan, etc.) selecting the "Pages from the UK" option to limit results to country based sites? Or if not hard stats, at least any international users care to comment? Cheers, Brian ~identity
International SEO | | identity0 -
My blog not appear in google today
hi i have a small blog on ( blogger/blogspot ) with more than 5k/day visitors it was doing well in google through 9 months and was appear in more than 2000 keywords but ,today when check google i found my blog not appear in any keyword !!!!! when i put my blog URL in google i found it ,but when i searched for any post title it not appear !!! i not changed any thing ,so it is a penalty from google ? and why ? thanks penalty
International SEO | | activeacts0 -
Is duplicate content really an issue on different International Google engines?
i.e. Google.com v.s. Google.co.uk This relates to another question I have open on a similar issue. So if I open the same e-commerce site (virtually) on company.com and company.co.uk, does Google really view that as duplicate content? I would be inclined to think they have that figured out but I havent had much experience with international SEO...
International SEO | | BlinkWeb0