Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's the difference between 'en-gb' and 'en-uk; when choosing Search engines in campaign set up?
Hi What's the difference search engine wise and which one should I choose, i presume GB since covers entire British landmass whereas UK excludes Ireland according to political definition, is it the same according to Google (& other engines) ? All Best Dan
International SEO | | Dan-Lawrence0 -
Google Webmaster showing error for [hreflang='x-default']
Hi There! Using [hreflang='x-default'] tag to target language specific countries on our site but Google Webmaster showing errors even implementation made as per Google guideline but one thing is not clear and we are not sure, this is the reason behind it. Error is showing up only on those pages where 'Google Parameters' are used. For example : https://www.sitegeek.com/a2hosting?grank=open 'grank=' is defined as a 'Google Parameters' and on the above page 'hreflang' tags are : Also, on page https://www.sitegeek.com/a2hosting [without Google Parameters] same above 'hreflang' tags are taken. But, There is no error on second page URL where no 'Google Parameters' in URL. Therefore, error showing on first URL where 'Google Parameters' are taken. Is this the issue or not? suggest how to remove? -- Rajiv S9vhl3T
International SEO | | gamesecure0 -
Google is still indexing with https,i removed ssl for my website
My website is claydip.com. I removed ssl for my website, but when i type claydip in google search it is still displaying with https and saying no description available..i lost visitors from search..kindly help me. I moved from bluehost to deamhost. I had a ssl at bluehost, when i move to dreamhost i am not using it.
International SEO | | knextweb8190 -
Country subfolders showing as sitelinks in Google, country targeting for home page no longer working
Hi There, Just wondering if you can help. Our site has 3 region versions (General .com, /ie/ for Ireland and /gb/ for UK), each submitted to Google Webmaster Tools as seperate sites with hreflang tags in the head section of all pages. Google was showing the correct results for a few weeks, but I resubmitted the home pages with slight text changes last week and something strange happened, though it may have been coincidental timing. When we search for the brand name in google.ie or google.co.uk, the .com now shows as the main site, where the sitelinks still show the correct country versions. However, the country subdirectories are now appearing as sitelinks, which is likely causing the problem. I have demoted these on GWT, but unsure as to whether that will work and it seems to take a while for sitelink demotion to work. Has anyone had anything similar happen? I thought perhaps it was a markup issue breaking the head section so that Google can no longer see the hreflangs pointing to each other as alternates. I checked the source code in w3 validator and it doesn't show any errors. Anyway, any help would be much appreciated - and thanks to anyone who gets back, it's a tricky type of issue to troubleshoot. Thanks, Ro
International SEO | | romh0 -
How can I change the currency Google lists my products with in the SERP?
I.e. This product - http://www.absoluteautomation.ca/fgd400-sensaphone400-p/fgd400.htm - shows up as USD in the SERP. (In the US it just won't show a currency, if Canada it will show USD on the SERP). My pricing is all in CAD, how can I tell Google this? (It knows pricing is CAD in my Google Product Listings/Merchant Center). Thanks!
International SEO | | absoauto0 -
Thai Characters in URL's
Does anyone have experience with non-Latin characters in URL's? We've launched a website in Thailand and picked Thai characters for URL's. However, when you copy it, it turns into something like this: http://www.imoneythailand.com/บัตรเครดิต Can it impact our website's crawlability? Also, is keyword in URL a ranking factor for non-Latin languages? Thanks in advance for help!
International SEO | | imoney0 -
Google search cache points to and uses content from different url
We have two sites, 1 in new zealand: ecostore.co.nz and 1 in Australia: ecostoreaustralia.com.au Both sites have been assigned with the correct country in Webmaster tools Both site use the same urls structure and content for product and category pages Both sites run off the same server in the US but have unique ip adresses. When I go to google.com.au and search for: site:ecostoreaustralia.com.au I get results which google says are from the Australian domain yet on closer inspection it is actually drawing content from the NZ website. When I view a cached page the URL bar displays the AU domain name but on the page (in the top grey box) it says: _This is Google's cache of http://www.ecostore.co.nz/pages/our-highlights. _ Here is the link to this page: http://webcache.googleusercontent.com/search?q=cache:Zg_CYkqyjP4J:www.ecostoreaustralia.com.au/pages/our-highlights+&cd=7&hl=en&ct=clnk&gl=au In the last four weeks the ranking of the AU website has dropped significantly and the NZ site now ranks first in Google AU, where before the AU site was listed first. Any idea what is going wrong here?
International SEO | | ArchMedia0 -
Does publishing your physical company address increase your Google Trust?
The question is in the title: Does publishing your physical company address increase your Google Trust? I.e. on the contact page of the website, providing a physical address. The address would match the WHOIS entry for the domain, and allow customers to contact you via post.
International SEO | | Peter2640