What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How should I handle hreflang tags if it's the same language in all targeting countries?
My company is creating an international version of our site at international.example.com. We are located in the US with our main site at www.example.com targeting US & Canada but offering slightly different products elsewhere internationally. Ideally, we would have hreflang tags for different versions in different languages, however, it's going to be an almost duplicate site besides a few different SKUs. All language and content on the site is going to be in English. Again, the only content changing is slightly different SKUs, they are almost identical sites. The subdomain is our only option right now. Should we implement hreflang tags even if both languages are English and only some of the content is different? Or will having just canonicals be fine? How should we handle this? Would it make sense to use hreflang this way and include it on both versions? I believe this would be signaling for US & Canda visitors to visit our main site and all other users go to the international site. Am I thinking this correctly or should we be doing this a different way?
International SEO | | tcope250 -
Google Search Console "International Targeting" is reporting errors that are not present on my site
We are currently handling search for a global brand www.example.com/ which has presence in many countries worldwide. To help Google understand that there is an alternate version of the website available in another language, we have used hreflang tags. These hreflang tags are implemented only via the XML sitemap across all geo-locations. Under the “Search Analytics -> International Targeting” section, in Google Search Console, for the Malaysian website (www.example.com/my/), there are a number of “no-return tags (sitemaps)” errors arising. For example, for India as a geo-location, there is one ‘en-IN’ – no return tags (sitemaps) errors listed. The error is listed below: Originating URL - www.example.com/my/xyz/ Alternate URL - www.example.com/in/xyz/ When the XML sitemap for the URL – www.example.com/in/ was checked for the hreflang tags, it was noticed that the implementation of hreflang tags for the URL – www.example.com/in/xyz/ was perfectly fine and it was providing a return tag to the URL – www.example.com/my/xyz/. After the code level verification, it was identified that the implementation of hreflang tags was perfectly fine via the XML sitemap. Even though at the code level it was verified that the implementation is fine, the error still persists in Google Search Console. Kindly suggest a solution to this situation, and also advise the effects of these errors on search engine performance
International SEO | | Starcom_Search0 -
Setting up I.P Filter Google Analytics - I.p ending with 0/24
Hi everyone, Your help would be much appreciated for the following: I am trying to setup I.P filters for our Google Analytic account to exclude internal traffic. We are located in multiple locations and each location have multiple I.p addresses. The I.P addresses we have end either by 0/24 which apparently means they provide a range from 0 to 255 and or 128/25. I have tried to setup the I.P addresses in different formats on the GA filter but they are apparently are not valid: example of one setup I tried: 1**.\2**.\8*.([0-256]) I have gone through the Filter setup guide from Google but I must be doing something wrong- probably to do on how I setup the I.P's ending with 0/24 and 128/25 If anyone could help me on how I can set up the I.P filters Google analytic would be great. The I.P addresses look like the following (changed digits): Location 1: 174.177.179.0/25 174.177.179.128/25 Location 2: 196.222.87.0/24
International SEO | | AlphaDigital2
194.59.197.0/24 Thanks you so much for your help, L.0 -
Blocking domestic Google's in Robots.txt
Hey, I want to block Google.co.uk from crawling a site but want Google.de to crawl it. I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers? any ideas? Thanks B
International SEO | | Bush_JSM0 -
Does Google take into account the place where the server is hosted to rank the pages.
Does Google take into account the place where the server is hosted to rank the pages. What I mean is, if I have a server in USA and I am working for the Spain marketplace: Will Google rank better my pages for this market if the server were hosted in Spain?
International SEO | | NorbertoMM0 -
Ranking issues for UK vs US spelling - advice please
Hi guys, I'm reaching out here for what may seem to be a very simple and obvious issue, but not something I can find a good answer for. We have a .com site hosted in Germany that serves our worldwide audience. The site is in English, but our business language is British (UK) English. This means that we rank very well for (e.g.) optimisation software but optimization software is nowhere to be found. The cause of this to me seems obvious; a robot reading those two phrases sees two distinct words. Nonetheless, having seen discussions of a similar nature around the use of plurals in keywords, it would seem to me that Google should have this sort of thing covered. Am I right or wrong here? If I'm wrong, then what are my options? I really don't want to have to make a copy of the entire site; apart from the additional effort involved in content upkeep I see this path fraught with duplicate content issues. Any help is very much appreciated, thanks.
International SEO | | StevenHowe0 -
CcTLD not ranking in other country.
I'm trying to rank a .im domain in Australia and having a hard time getting any traction at all. GWT won't let me geo-target the domain for Australia. It is currently set for Isle of Man. Thinking of registering a .com.au (same domain isn't available, but I can get something close) and mirror the site completely. I'd canonical all urls on the .com.au to the corresponding .im pages - would this make a difference? Also, since the linkbuilding is being done for the .im domain - I think I'll have to start over on the .com.au domain. Thoughts, tips and insight highly appreciated mozzers!
International SEO | | bobjones0 -
Non US site pages indexed in US Google search
Hi, We are having a global site wide issue with non US site pages being indexed by Google and served up in US search results. Conversley, we have US en pages showing in the Japan Google search results. We currently us IP detect to direct users to the correct regional site but it isn't effective if the users are entering through an incorrect regional page. At the top of each or our pages we have a drop down menu to allow users to manually select their preferred region. Is it possible that Google Bot is crawling these links and indexing these other regional pages as US and not detecting it due to our URL structure? Below are examples of two of our URLs for reference - one from Canada, the other from the US /ca/en/prod4130078/2500058/catalog50008/ /us/en/prod4130078/2500058/catalog20038/ If that is, in fact, what is happening, would setting the links within the drop down to 'no follow' address the problem? Thank you. Angie
International SEO | | Corel0