Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problem to get multilingual posts indexed on Google
Last year on June I decided to make my site multi-lingual. The domain is: https://www.dailyblogprofits.com/ The main language English and I added Portuguese and a few posts on Spanish. What happened since then? I started losing traffic from Google and posts on Portuguese are not being indexed. I use WPML plugin to make it multi-lingual and I had Yoast installed. This week I uninstalled Yoast and when I type on google "site:site:dailyblogprofits.com/pt-br" I started seeing Google indexing images, but still not the missing posts. I have around 145 posts on Portuguese, but on Search Console it show only 57 hreflang tags. Any idea what is the problem? I'm willing to pay for an SEO Expert to resolve this problem to me.
International SEO | | Cleber0090 -
Google.ie returning more and more UK based results, why?
I have discovered the most infuriating issue with Google Search for Irish users and it seems to be getting increasingly worse in the last 2 years or so. This is not only frustrating as a business owner (in fact it could bring a business to its knees) but it is rage inducing as a consumer.
International SEO | | Secrets
Google knows the location where I am searching from and I'm using google.ie yet I still get just a small number of Irish websites usually followed by eBay and Amazon results then a never ending list of websites that are based in the United Kingdom. Now, I know the one thing that we all have in common is the use of the English language, however what we don't have in common is shipping costs. In order to slightly increase the number of Irish based companies I need to add in the phrase 'Ireland' to my search (on google.ie in Ireland) and this makes only a small difference. In fact, oftentimes Google seems to throw in the odd American or Australian site just to really wind me up.
It's completely absurd that Google rarely returns results for .ie websites or irish based websites when searching in Ireland. Many UK companies don't ship to Ireland (including many of the eBay and Amazon results). This is killing Irish businesses who have the products and cheaper or free shipping and many how are working damn hard on their SEO are still being passed up for companies that have nothing to do with our economy.... Why oh why is this happening.0 -
E-Commerce - Country Domains versus 1 Domain?
Hi, Just wanted to get some feedback and opinions re the idea of segmenting our ecommerce site languages under various domains, like .jp for japanese, .it for italian etc... I do understand the geolocation benefits that this could bring us, but on the flipside, it would mean that we would need to grow our domain authority, link buiding per country domain, which is quite a bit of work. Has anyone ever considered or implemented this and any thoughts? Thanks!
International SEO | | bjs20100 -
If I redirect based on IP will Google still crawl my international sites if I implement Hreflang
We are setting up several international sites. Ideally, we wouldn't set up any redirects, but if we have to (for merchandising reasons etc) I'd like to assess what the next best option would be. A secondary option could be that we implement the redirects based on IP. However, Google then wouldn't be able to access the content for all the international sites (we're setting up 6 in total) and would only index the .com site. I'm wondering whether the Hreflang annotations would still allow Google to find the International sites? If not, that's a lot of content we are not fully benefiting from. Another option could be that we treat the Googlebot user agent differently, but this would probably be considered as cloaking by the G-Man. If there are any other options, please let me know.
International SEO | | Ben.JD0 -
Geographic Target set up in Google webmaster tool
Hi, When i launched my web site 3 months ago ( I'm am very new to SEO) I have set up the geographical target section in Google webmaster tool for US. Now, I'm thinking to change it to some other geo target to see if i can get more traffic. However, recently few of my prompted keywords got really well in Google US ranking. Here are my Questions: if i will change the geo settings in webmaster tool will effect the ranking i already managed to achieve in US? In the list of all the countries in Google webmaster tools what does is mean "unlisted"? Can i select more than one country to target and if I can how? Thanks!! Raviv
International SEO | | Indiatravelz0 -
Website Target in Europe
Hi, I am planning a site to target in Europe and I expect to translate my site into ten different languages namely English, French, Spanish, Italian, German, Russian, Greek, Portuguese, Dutch and Swedish. I am doing some study of this case in targeting different countries for SEO, most of the advise are the following: a. Build 10 different websites and target different geographical location in Google Webmaster b. Get 10 different country specific domains for 10 different websites I would like to hear any suggestion if there is anything better than this ? I had all the materials and translation ready but building 10 different websites or getting 10 different domains are very time consuming and costly. I would be appreciated if any one had any advise for me to make the website more management friendly. Thank you. Tom
International SEO | | Stevejobs20110 -
Non US site pages indexed in US Google search
Hi, We are having a global site wide issue with non US site pages being indexed by Google and served up in US search results. Conversley, we have US en pages showing in the Japan Google search results. We currently us IP detect to direct users to the correct regional site but it isn't effective if the users are entering through an incorrect regional page. At the top of each or our pages we have a drop down menu to allow users to manually select their preferred region. Is it possible that Google Bot is crawling these links and indexing these other regional pages as US and not detecting it due to our URL structure? Below are examples of two of our URLs for reference - one from Canada, the other from the US /ca/en/prod4130078/2500058/catalog50008/ /us/en/prod4130078/2500058/catalog20038/ If that is, in fact, what is happening, would setting the links within the drop down to 'no follow' address the problem? Thank you. Angie
International SEO | | Corel0 -
Targeting Different Countries... One Site or Separate?
I have a client who has 3 ecommerce sites. They are somewhat differentiated but for the most part sell the same stuff. Luckily 2 of them are quite authoritative, old and rank reasonably well. Most of the visitors and sales come from the US. He wants to start targeting Europe, Mexico and Canada. What are your suggestions for doing this? Are we better targeting on the main domains? Not really sure how to do that? Should we use a subdomain and a new store front for each geo? Should we use a .co.uk .co.mx and .co.ca each with a unique storefront? It looks like we are moving to a Magento platform so setting up multiple storefronts on a single database is not a big issue. Anyone have any experience with this?
International SEO | | BlinkWeb0