Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which Google does a .ie website get shown in?
Hi all, I am working on a .ie website and I was under the understanding that if you have a regional domain, like .ie you will limited to being shown in a search engine like google.ie When I go to International Targeting in Google SEarch Console it says the site is associated with: Ireland Am I limiting my ability to rank well in worldwide Google searches with this domain and if so, how can I counter this? Many thanks.
International SEO | | Bee1590 -
Can you target the same site with multiple country HREFlang entries?
Hi, I have a question regarding the country targeting aspect of HREFLANG. Can the same site be targeted with multiple country HREFlang entries? Example: A global company has an English South African site (geotargeted in webmaster tools to South Africa), with a hreflang entry targeted to "en-za", to signify English language and South Africa as the country. Could you add entries to the same site to target other English speaking South African countries? Entries would look something like this: (cd = Congo, a completely random example) etc... Since you can only geo-target a site to one country in WMT would this be a viable option? Thanks in advance for any help! Vince
International SEO | | SimonByrneIFS0 -
How To Rank A UK Website On Google.com (US)
Hi, I've done some research on this but couldn't find any definitive answer I can trust! We have a client who resides in the UK. They have '.com' domain, hosted on a UK server, using UK spelling. Their business objective for this year is to expand in the USA, including the opening of a warehouse over there. They are wanting us to rank their website on both Google.co.uk and Google.com (North America); besides changing the geolocation settings in GWT's, and building links from .com websites is there anything else we can do to increase their visibility on Google.com? Many thanks in advance, appreciated!
International SEO | | Webpresence
Lee.0 -
If I redirect based on IP will Google still crawl my international sites if I implement Hreflang
We are setting up several international sites. Ideally, we wouldn't set up any redirects, but if we have to (for merchandising reasons etc) I'd like to assess what the next best option would be. A secondary option could be that we implement the redirects based on IP. However, Google then wouldn't be able to access the content for all the international sites (we're setting up 6 in total) and would only index the .com site. I'm wondering whether the Hreflang annotations would still allow Google to find the International sites? If not, that's a lot of content we are not fully benefiting from. Another option could be that we treat the Googlebot user agent differently, but this would probably be considered as cloaking by the G-Man. If there are any other options, please let me know.
International SEO | | Ben.JD0 -
How to fix the duplicate content problem on different domains (.nl /.be) of your brand's websites in multiple countries?
Dear all, what is the best way to fix the duplicate content problem on different domains (.nl /.be) of your brand's websites in multiple countries? What must I add to my code of websites my .nl domain to avoid duplicate content and to keep the .nl website out of google.be, but still well-indexed in google.nl? What must I add to my code of websites my .be domain to avoid duplicate content and to keep the .nl website out of google.be, but still well-indexed in google.nl? Thanks in advance!
International SEO | | HMK-NL3 -
IP Redirection vs. cloaking: no clear directives from Google
Hi there, Here is our situation:we need to force an IP Redirection for our US users to www.domain.com and at the same time we have different country-specific subfolders with thei own language such as www.domain.com/fr. Our fear is that by forcing an IP redirection for US IP, we will prevent googlebot (which has an US IP) from crawling our country-specific subfolders. I didn't find any clear directives from Google representatives on that matter. In this video Matt Cutts says it's always better to show Googlebot the same content as your users http://www.youtube.com/watch?v=GFf1gwr6HJw&noredirect=1, but on the other hand in that other video he says "Google basically crawls from one IP address range worldwide because (they) have one index worldwide. (They) don't build different indices, one for each country". This seems a contradiction to me... Thank you for your help !! Matteo
International SEO | | H-FARM0 -
Country specific domains pointing to a .com site
Hello, I am new to seo so please be easy if this happens to be a "silly" question. My company has a .com site. We are expanding into global markets, focusing on specific countries right now. General question: Would I be penalized for duplicate content if I purchased country-specific domains and pointed them to the .com site? Thanks, Jim
International SEO | | jimmer0 -
Subdomain hosted in a different country - what are the implications?
Hello, We are looking at creating an eCommerce section to a website and we are just weighing up the options: Magento - host on hour own server - great but it can often be very slow when hosting a shared server. Shopify - hosted solution but hosting is in the US and we are in the UK and shop will be hosted on a subdomain as a result Build our own solution - time consuming and costly There are two issues that have arisen from this situation.... Is it worse for SEO to host your store in a different country or to host in your country but your store potentially run slower? I'm swaying to the side of the argument that says give your users a good and fast experience instead of worrying about where you host the store. Bearing in mind that the main website will be hosted in the UK anyway and it is just the subdomain that will be hosted in the US. Just wondered if anybody has had experience with this or if I'm missing something? All feedback greatly appreciated! Thanks, Elias
International SEO | | A_Q0