Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console International Targeting - Works for Hungary, but not Ireland - Why?
company.com (root)
International SEO | | scottclark
USA - lang="en" | GSC target: USA (shows US site in SERPs for "companyname" search)
company.com/hungary
Hungary - lang="hu" | GSC target: Hungary (shows Hungarian site in SERPs for "companyname" search)
AWESOMENESS company.com/ireland
Ireland - lang="en" | GSC target: Ireland (shows US site (doh!) in SERPs "companyname" search)
NOT RIGHT! It is our theory [please weigh in!] that because we don't have a company.com/usa folder, the TLD targeting (EN) is overriding other English language sites in some manner. In other words, the reason it's not overriding Hungary is because it's a different language. What must we do to get the Irish site ranked for "companyname" searches and to show by default in Ireland?0 -
What is the best way to generate an automatic sitemap for google, bing and yahoo?
Hi guys, I have 3 international sites So far I recieved a Sitemap that was generated by an seo company to use and submit to google, for our co.nz domain, I have been told to submit this also to bing and yahoo. Can anyone tell me if I can submit the same sitemap for the com.au and com or would i need to generate a new sitemap for each domain? I have been told, everytime we change content we have to keep submitting, is there a way to do this autmatically as we will be writting alot of content daily. Any recommendations or suggestions?
International SEO | | edward-may0 -
Does Google's algorithm work the same in every country?
I can't help but feel this is a silly question! but does Google algorithm work exactly the same throughout all countries? I run a few sites in the UK and a couple in Spain but can't help but feel that my Spanish sites are harder to rank for. The sites that rank the best are business directories in Spain... whereas here in the UK you'd be lucky to find one on page one..
International SEO | | david.smith.segarra0 -
How to make Google consider my international subdomain relevant?
We have recently started to look deeper into international SEO. We have search engine optimized our international landing pages, title tags and meta descriptions with keywords etc. so each of the international language we support is SEO'ed for the local market. We support 12 languages, and each of them are located on a subdomain. That means if we say our site is helloworld.com, a person from Germany that lands on this site can switch to German and will then be redirected to de.helloworld.com and all content will be in German. Our problem is that we develop cloud-based software, we have a significant amount of traffic, but whenever we get media coverage or people link to us from anywhere in the world they always link to the root domain which in this case then would be helloworld.com. That means if I go to google.de and type in the exact meta description or title tag we use in German, the Google search engine can't even find us because "I assume" Google don't consider our de.helloworld.com relevant because nobody has ever linked to this site. I would appreciate very much if anyone can give me some advice on how I can address this issue. Thanks a lot! Allan
International SEO | | Todoist0 -
Can I add more than one XML sitemap in my Robots.txt File?
I have 3 domains all hosted on the same Magento Enterprise Platform. Each domain targets a different county. .co.uk, .com.au, and .com. Can I add all three sitemaps in my robots.txt file? www.example.com/sitemap.xml www.example.com.au/sitemap.xml www.example.co.uk/sitemap.xml
International SEO | | hfranz0 -
Non US site pages indexed in US Google search
Hi, We are having a global site wide issue with non US site pages being indexed by Google and served up in US search results. Conversley, we have US en pages showing in the Japan Google search results. We currently us IP detect to direct users to the correct regional site but it isn't effective if the users are entering through an incorrect regional page. At the top of each or our pages we have a drop down menu to allow users to manually select their preferred region. Is it possible that Google Bot is crawling these links and indexing these other regional pages as US and not detecting it due to our URL structure? Below are examples of two of our URLs for reference - one from Canada, the other from the US /ca/en/prod4130078/2500058/catalog50008/ /us/en/prod4130078/2500058/catalog20038/ If that is, in fact, what is happening, would setting the links within the drop down to 'no follow' address the problem? Thank you. Angie
International SEO | | Corel0 -
Ranking well in Google US but not showing up in UK
Hi, We are trying to position a specific site for a big sport event that will be played at the end of May. We only care about Google UK. It had no relevant content until last week. The site is a almost a year old but we just started to get links for it (all from the UK). They worked so fast: we were in page 2 of SERPs in the UK within a few days. But suddenly the site has disappeared and we're not even in top 100 anymore. However, we are ranking extremely well in the US (first page for the keywords we wanted to rank for). The site is a .com and it's hosted in Wordpress.com (with a custom URL). I understand that that can be a problem, but we have already told Google that our business is in the UK through Webmaster Tools. I guess that with time and more UK links, Google would ultimately understand that our focus is the UK audience, but unfortunately we have no time and we can't wait due to the date of the sport event. What do you think that we can do to rank well in the UK and not in the US as fast as possible? Thanks!
International SEO | | jorgediaz0 -
What's the best strategy for checking international rankings?
Hi There- I am looking to optimize sites serving the UK and Austrailia markets. I feel like I have a good handle on how to go about doing that, but what I am fuzzy on is, what's the best way to monitor the SERPs for the keywords I am targeting. I know based on experience that if I just search google.com.au from here in the states, my results will be 'americanized' and may/probably won't accurately reflect what someone would see if they were search from Austrailia. Are there any good tools or tactics for seeing what searchers in the countries I am focusing on woudl see? Thanks! Jason
International SEO | | phantom0