What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?

[Envoke-Marketing]

Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?

[Hannah_Smith]

Hi Keith,

In my experience Google mainly crawls from the US.

You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.

As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.

This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.

I hope this helps,

Hannah

[FXDD]

Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this.  Hopefully this post gets freshly reviewed

[WendyWuTours]

Interesting question - I'd quite like to know what happens here too.

Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).

QHtnfOgp65Q

[Envoke-Marketing]

Hi,

The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.

To clear up the second question, let me be more specific:

Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?

[Gyorgy]

I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9

Your other question is a bit confusing. Why Google wouldn't see the UK site?