Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to check Robots.txt File
I want to know that how we are going to check that the Robots.txt File of the website is working properly. Kindly elaborate the mechanism for it please.
International SEO | | seobac1 -
If domain mapping subfolders to TLD's is it perceived as a fully separate entity/site therafter ?
Hi I take it once you have domain mapped a country specific subfolder to a country specific TLD (for better local region targeting reasons) Google perceives it as a completely separate entity and it no longer shares any of the parent sites domain benefits (such as domain authority etc) so from that point on requires its own dedicated link building etc ? All Best Dan
International SEO | | Dan-Lawrence0 -
Ranking UK company in Google.com
Hi all, I have a UK client with a .com domain, hosted on a US server, but the physical business premises is based in the UK. Their product is a really great product and available for export to the US. I want to rank them higher in the US, more specifically Google.com. I've helped them rank very well organically in the UK (google.co.uk) for some great terms, however they rank almost nowhere in google.com (gl=us) for the same terms, for example: In Google.co.uk they rank #3 for the key-phrase.
International SEO | | seowoody
In Google.com they rank #90 for the same key-phrase. I've got them some great US focused links with PR coverage including MSN Cars, nydailynews.com etc. I just wondered if there was any one "golden ticket" for boosting US rankings? I've read that a physical business premises located in the US helps a lot. Can anyone confirm this and if so, would a rented PO box in the US help? The site has great social signals too, growing twitter following and many FB likes/shares etc. Any other tips/advice? Thanks in advance,
Cheers,
Woody 🙂0 -
How does Google Serve the correct language version?
Hi guys, I'm currently working on a multi lingual .eu website with 20 lanuages which has been live now for a month now. Its is a sub directory set up so the french language version would look something like www.example.eu/fr. The English language version defaults to the home page www.example.eu. When you search for the brand name on the local search engines the home page English version appears instead of the preferred language version for each Country. 90% of the external links so far link to the homepage so I'm guessing this is the reason why. We are still waiting for the development company to create sitemap for each language using the rel=”alternate” hreflang=”x” XML Sitemap Tool. I know Google look at a number of factors when deciding what results to give a user. Can anybody share their experience or advice here? Thanks Rob
International SEO | | daracreative0 -
Geo Targeting SEO Techniques for Google UK
I'm starting a new SEO project whereby I'll be targeting UK search engines only such as Google.co.uk, (I'm from the states) and I'm gathering all the information I can get on this topic Obviously, I got a CO.UK TLD, and hosting/IP is UK based, but can anyone shed light on other techniques that has worked for you, Besides of the above here is some advice I picked up so far; Regional directory listings,
International SEO | | Plorex
Inbound and outbound inks from/to UK based websites,
Geographic targeting in Google webmaster tools,
British slang... What else is there?
Much appreciated0 -
How do you submit US based news on Google News US from the UK?
How do you go about submitting US news (based in the UK) to get indexed on Google News and show up in the US rather then the UK?
International SEO | | CameronT0 -
What's the best strategy for checking international rankings?
Hi There- I am looking to optimize sites serving the UK and Austrailia markets. I feel like I have a good handle on how to go about doing that, but what I am fuzzy on is, what's the best way to monitor the SERPs for the keywords I am targeting. I know based on experience that if I just search google.com.au from here in the states, my results will be 'americanized' and may/probably won't accurately reflect what someone would see if they were search from Austrailia. Are there any good tools or tactics for seeing what searchers in the countries I am focusing on woudl see? Thanks! Jason
International SEO | | phantom0 -
Google Webmaster Tools - International SEO Geo-Targeting site with Worldwide rankings
I have a client who already has rankings in the US & internationally. The site is broken down like this: url.com (main site with USA & International Rankings) url.com/de url.com/de-english url.com/ng url.com/au url.com/ch url.com/ch-french url.com/etc Each folder has it's own sitmap & relative content for it's respective country. I am reading in google webmaster tools > site config > settings, the option under 'Learn More': "If you don't want your site associated with any location, select Unlisted." If I want to keep my client's international rankings the way it currently is on url.com, do NOT geo target to United States? So I select unlisted, right? Would I use geo targeting on the url.com/de, url.com/de-english, url.com/ng, url.com/au and so on?
International SEO | | Francisco_Meza0