What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Soft Launch App with Country Targeting To Match?
Hi,
International SEO | | eTinaRose
We're gearing up to do a soft launch of our app in a few countries, such as Australia. Our website says to request early access since the app is not yet publicly available in the apple store. However, it will be in those soft launch countries. This is what I'm considering... Creating a subdomain site such as au.liquidtext.net (liquidtext.au is not available). And setting that site, au.liquidtext.net, to be country targeted to Australia via webmaster tools. Once the app is available internationally, and the main site has been updated to show this, I would then redirect any subdomains to the main site. Thoughts? Are there any negative repercussions to this this approach that I am not considering? Thank you for your help Moz community!! Tina0 -
Multi-Country Duplicate Content
Hello, We have an ecommerce site that serves several countries on the same .com domain - US, UK and CA. We have duplicate content across these countries because they are all English speaking so there is little variance in the pages and they each sell most of the same products. We have implemented hreflang into our sitemaps but we need to address the duplicate content. We were advised to canonicalize our UK and CA pages back to the duplicate US pages (our US pages account for the majority of our traffic and sales). This would cause the UK and CA pages to fall out of the index but the visitor would still be taken to the correct country's page due to the hreflang. I'm leary about doing this because they are across countries. Is this ok to do? If not, how do we address the duplicate content since they are not on their own CCTLD's?
International SEO | | Colbys0 -
Sub-domains or sub-directories for country-specific versions of the site?
What approach do you think would be better from an SEO perspective when creating country-targeted versions for an eCommerce site (all in the same language with slight regional changes) - sub-domains or sub-directories? Is any of the approaches more cost effective, web development-wise? I know this topic's been under much debate and I would really like to hear your opinion. Many thanks!
International SEO | | ramarketing0 -
Website Target in Europe
Hi, I am planning a site to target in Europe and I expect to translate my site into ten different languages namely English, French, Spanish, Italian, German, Russian, Greek, Portuguese, Dutch and Swedish. I am doing some study of this case in targeting different countries for SEO, most of the advise are the following: a. Build 10 different websites and target different geographical location in Google Webmaster b. Get 10 different country specific domains for 10 different websites I would like to hear any suggestion if there is anything better than this ? I had all the materials and translation ready but building 10 different websites or getting 10 different domains are very time consuming and costly. I would be appreciated if any one had any advise for me to make the website more management friendly. Thank you. Tom
International SEO | | Stevejobs20110 -
How can I replicate India SERPs from US
I was recently checking out our analytics keywords stats and I am seeing traffic noted as google organic traffic coming from India. When I do a search on http://www.google.co.in/ for those terms using a "incognito" browser with no cookies, and not logged in I don't see our site in the first 100 results. The same thing happens on google.com if I try to set the account in india. I am trying to see if this might be an issue with an adwords campaign not getting auto-tagged (auto-tagging is turned on in adwords), or if there is something else going on. The keyword campaign gets many more clicks than analytics shows for this organic option so perhaps it is an odd rare occurrence. I am glad to have the traffic, but I want to be sure it is actually tracking correctly as organic, or if PPC is not tagging for some reason 😉 Any suggestions or thoughts?
International SEO | | SL_SEM0 -
Www.google.rs for SERBIA is mising in Engienes
Why you dont have SERBIA in engines for google - www.google.rs You have Croatia, Bosnia and Hercegovina and Slovenia and Serbia is missing 😞 Best regards kasa
International SEO | | stamparija2 -
How can I view Google.com SERPs from outside the US?
If I go to Google.com I get redirected back to Google.co.uk and search results have a UK bias. I'm trying to research the US market and have a hazy recolection of Rand demonstrating how you can add a few characters to the google.co.uk url to see US results - just can't remember what video I saw it in. Any ideas? Thanks a lot
International SEO | | trbaldwin0