Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How To Rank A UK Website On Google.com (US)
Hi, I've done some research on this but couldn't find any definitive answer I can trust! We have a client who resides in the UK. They have '.com' domain, hosted on a UK server, using UK spelling. Their business objective for this year is to expand in the USA, including the opening of a warehouse over there. They are wanting us to rank their website on both Google.co.uk and Google.com (North America); besides changing the geolocation settings in GWT's, and building links from .com websites is there anything else we can do to increase their visibility on Google.com? Many thanks in advance, appreciated!
International SEO | | Webpresence
Lee.0 -
What is best practice of using google translate
Hi, I'm thinking of adding google translate to our retailing site so that we could reach more international customers. what about the pros&cons? Any experience of success of utilising it and what potential issues should I be looking at? Thanks
International SEO | | LauraHT0 -
Getting pages that load dynamically into the SE's
SEO'ers, Am dealing with an issue I cannot figure out the best way to handle. Working on a website that shows the definitions of words which are loaded dynamically from an open source. Source such as: wiktionary.org When you visit a particular page to see the definition of the word, say; www.example.com/dictionary/example/ the definition is there. However, how can we get all the definition pages to get indexed in search engines? The WordPress sitemap plugin is not picking up these pages to be added automatically - guess because it's dynamic - but when using a sitemap crawler pages are detected. Can anybody give advice on how to go about getting the 200k+ pages indexed in the SE's? If it helps, here's a reference site that seems to load it's definitions dynamically and has succeeded in getting its pages indexed: http://www.encyclo.nl/begrip/sample
International SEO | | RonFav0 -
What language to use for URL's for Russian language?
Hi, Our site is in English, Spanish, Danish and Russian - the URL's are individual to the language they are in, but of course, Russian contains some strange characters so I decided not to use them in the URL's Any advice on how to create the URL's for russian language pages? thanks
International SEO | | bjs20100 -
Best practice for WPML, Yoast and sitemap for google
Hi Mozzers, I have a wordpress installation and work on a hotel website in three different languages: English, German, Spanish. In order to manage each language as a regional or global website, I started to give the website the names like: de.hotelnamen.com, es.hotelname.com (Hotel is in Costa Rica, maybe cr.hotelname.com is even better???) and hotelname.com. The possibility of WPML to manage my multilingual blog is good and the Yoast plugin gives me the sitemaps I want for each language. Because it is a hotel I have to have a global page which should serve the world if they try to find the hotel, right? That's why i put hotelname.com as a global page in English and registered the sitemaps and page on my webmaster account in countries as "not listed". For de.hotelname.com I choose Germany on another webmaster account and for es.hotelname.com Costa Rica (the country in which the Hotel is located). Unfortunately, after three month I don't receive good results with that methode. The hotelname.com adress is always the page which comes up in all search engines. I my tactic wrong? Where is my mistake? I would like to have hotelname.com in the rankings of all search engines beside of google Germany because for the German market I have the German version. Same in Costa Rica. Thanks for some ideas...
International SEO | | reisefm0 -
Poor Google.co.uk ranking for a UK based .net, but great Google.com
I run an extremely popular news & community website at http://www.onedirection.net, but we're having a few ranking issues in Google.co.uk. The site gets most of its traffic from the USA which isnt a bad thing - but for our key term "one direction", we currently don't rank at all on Google.co.uk. The site is located on a server based in Manchester, UK, and we used to rank very well earlier this year - fluttering about in position 5-7 most of the time. However earlier this year, around July, we started to fall down to page 2 or 3, and at the start of this month we don't rank at all for "one direction" on Google.co.uk. On Google.com however we're very strong, always on page one. We're definitely indexed on .co.uk, just not for main search term - which I find a bit frustrating. All the content on our site is unique, and we write 2-4 stories every day. We have an active forum too, so a lot of our content is user-generated. We've never had any "unnatural link building" messages in Webmaster Tools, and our link profile looks fine to me. Do we just need more .co.uk links, or are we being penalised for something? (I can't imagine what though). It certainly seems that way though. Another site, "www.onedirection.co.uk" which is never updated and has a blatant ad for something completely unrelated on its homepage, ranks above us at the moment- which I find quite frankly appalling as our site is pretty much regarded as the worlds most popular One Direction news and fan site. We've spent the last few months improving the page-load times of our site, and we've reduced any unneccesary internal linking on the site. Approx 2 months ago we launched a new forum on the site, 301'ing all the old forum links to the new one, so that could have had an impact on rankings - but we'd expect to see an impact on Google.com as well if this was an issue. We definitely feel that we should be ranking higher on Google.co.uk. Does anyone have any ideas what the iproblems could be? Cheers, Chris.
International SEO | | PixelKicks0 -
Why don't our English versions show up first?
If I google "greatfire" I find the Chinese version of our website (zh.greatfire.org) before the English version (en.greatfire.org). This is not on the Chinese-language version of Google. Why is this? Our site even has a language indicator () and also hints of where the English version is (). The same thing happens if I google "freeweibo". I find https://freeweibo.com but not https://freeweibo.com/en/, even though we indicate that's the English version (). Any ideas?
International SEO | | GreatFire.org0 -
Country name displayed after domain name in google SERP
our online shop targets clients in the US and worldwide (same URL - no subdirectories - currency changes based on IP). when searching in google.ie or google.no for our site google displays in the SERPS "US" or "United States" after the URL for our site, but for most other US competitors it does not show the country in the SERPS. I deleted our google places listing 2 weeks ago, since I suspected it may be related, but no change so far. In google webmaster tools we have targeted the shop domain to United States, which may be another factor. Unfortunately we can not undo this setting since without it our google US ranking for the most relevant competitive keyword drops from position 8 to position 100+. Server location is in Germany which despite lots of US links and US contact info and USD currency appparently makes google think that the site is not targeting the US. Does anybody know what triggers the country name in the SERPS (google places or webmaster tools or other) and can give advice if there is any way to get rid of it.
International SEO | | lcourse0