Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Google is not indexing each country/language subfolder on the ranks?
Hi folks, We use Magento 2 for the multi-country shops (its a multistore). The URL: www.avarcas.com The first days Google indexed the proper url in each country: avarcas.com/uk avarcas.com/de ... Some days later, all the countries are just indexing / (the root). I correctly set the subfolders in Webmaster tools. What's happening? Thanks
International SEO | | administratorwibee0 -
How can I change the currency Google lists my products with in the SERP?
I.e. This product - http://www.absoluteautomation.ca/fgd400-sensaphone400-p/fgd400.htm - shows up as USD in the SERP. (In the US it just won't show a currency, if Canada it will show USD on the SERP). My pricing is all in CAD, how can I tell Google this? (It knows pricing is CAD in my Google Product Listings/Merchant Center). Thanks!
International SEO | | absoauto0 -
Human Translation versus Google Translate for Ecommerce Products
Hi all, We want to put our products on our ecommerce site into another language. I have always been under the impression that running text through Google Translate is a no no, not only for the user experience, but also it is a Google tool and I am assuming that Google would notice that it is not translated by a human. I don't know if it would incur a penalty as such but it most likely would not be favoured as a human translation Can anyone confirm their experience or impression on this? Thanks!
International SEO | | bjs20100 -
Google Places and Hotel Finder
I work with a few hotels in Costa Rica and Nicaragua. Google Places has not launched in these regions so it has been impossible for me to start an account and get my properties listed in hotel finder. Somehow my competition has managed to get one of their properties listed and it's killing me! Does anyone know of any way to speed up the process? I'm very involved with Google+, FB, Twitter, Pinterest but G Places and Hotel finder or Google Maps don't want to pick me up. Any ideas? Thanks Mozzers. aSrDvhr
International SEO | | janders0 -
A google ranking problem
HI, Thought I would try SEOMoz out for a while and I am going to dive straight into my initial problem. I have my client moving up nicely in local Google search within my country but the problem is he seems to have actually slipped down a place when I search on google.com/ncr. (and on Rank Tracker) He does sell locally but is a worldwide supplier of his product and wants to obviously rank well everywhere. I understand why he has moved up locally but don't get the shift down internationally. So, 1. If everyone's Google is forcing local search results on everyone, how do you market better internationally. 2. Noticed that his competitors are all getting clever and now using keywords in their URL. Two guys are new on first page Google just because they have Keyworded urls. (killing me). The one guy has one backlink!! (although the client does have a keyworded URL stashed away but we don't want to got that route because we like to keep things 'white hat') The client is happy because when he searches he see's his site moving up nicely but I have a conscience and I am going to have to tell him whats going on. (that his Google is showing local results) I hope that all makes sense. Thanks Mike
International SEO | | MIkeCape0 -
Will Google punish me cuz my websites content are almost the same?
If I have almost the same contents for my three e-commerce websites, say A.com,B.uk,C.ca. They're promoted in US, GB, Canada which are all English speaking. Will my site be punished because they're almost the same to Google?
International SEO | | SquallPersun0 -
How do you submit US based news on Google News US from the UK?
How do you go about submitting US news (based in the UK) to get indexed on Google News and show up in the US rather then the UK?
International SEO | | CameronT0 -
My blog not appear in google today
hi i have a small blog on ( blogger/blogspot ) with more than 5k/day visitors it was doing well in google through 9 months and was appear in more than 2000 keywords but ,today when check google i found my blog not appear in any keyword !!!!! when i put my blog URL in google i found it ,but when i searched for any post title it not appear !!! i not changed any thing ,so it is a penalty from google ? and why ? thanks penalty
International SEO | | activeacts0