What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should Hreflang x-default be on every page of every country for an International company?
UPDATED 4/29/2019 4:33 PM I had made to many copy and pastes. Product pages are corrected Upon researching the hreflang x-default tag, I am getting some muddy results for implementation on an international company site older results say just homepage or the country selector but…. My Question/Direction going forward for the International Site I am working on: I believe I can to put x-default all the pages of every country and point it to the default language page for areas that are not covered with our current sites. Is this correct? From my internet reading, the x-default on every page is not truly necessary for Google but it will be valid implemented. My current site setup example:
International SEO | | gravymatt-se
https://www.bluewidgets.com Redirects to https://www.bluewidgets.com/us/en (functions as US/Global) Example Countries w/ code Site:- 4 countries/directories US/Global, France, Spain Would the code sample below be correct? https://www.bluewidgets.com/us/en/ (functions as US/Global) US/Global Country Homepage - https://www.bluewidgets.com/us/en/ US/Global Country Product Page(s) This would be for all products - https://www.bluewidgets.com/us/en/whizzer-5001/ http://www.bluewidgets.com/us/en (functions for France) France Country Homepage - https://www.bluewidgets.com/fr/fr/ France Country Product Page(s) This would be for all products- https://www.bluewidgets.com/es/es/whizzer-5001 http://www.bluewidgets.com/us/en (functions as Spain) Spain Country Homepage - https://www.bluewidgets.com/es/es/ Spain Country Product Page(s) This would be for all products - https://www.bluewidgets.com/es/es/whizzer-5001 Thanks for the spot check Gravy0 -
Is there a Risk Around Creating a Website for Each Country in The World?
Is there a risk around creating a website for each country in the world with similar to identical content depending on the language? We need to serve prices and the local currencies and be compliant with regulations. We're planning to use rel=canonical and HREFLANG tags to help with consolidation and GEO-targeting.
International SEO | | ari_seo0 -
How well does Google's "Locale-aware crawling by Googlebot" work?
Hello, In January of this year Google introduced "Locale-aware crawling by Googlebot." https://support.google.com/webmasters/answer/6144055?hl=e Google uses different crawl settings for sites that cannot have separate URLs for each locale. ......... This is basically for sites that dynamically render contend on the same URL depending on the locale and language (IP) of the visitor. If e.g. a visitor was coming from France, the targeted page would load in french. If a visitor was coming from the US the same page would load in English on the same URL. Does anyone have any experience with this setup and how well it works? How well do the different versions of a page get indexed, and how well do those pages rank? In the example above, does the french content get indexed correctly? Many thanks!
International SEO | | Veva0 -
What is the best way to generate an automatic sitemap for google, bing and yahoo?
Hi guys, I have 3 international sites So far I recieved a Sitemap that was generated by an seo company to use and submit to google, for our co.nz domain, I have been told to submit this also to bing and yahoo. Can anyone tell me if I can submit the same sitemap for the com.au and com or would i need to generate a new sitemap for each domain? I have been told, everytime we change content we have to keep submitting, is there a way to do this autmatically as we will be writting alot of content daily. Any recommendations or suggestions?
International SEO | | edward-may0 -
Google Analytics & Webmaster Tools Filtering
I've just set up a client who uses internationalization on Google Webmaster tools and Google Analytics. For easier management they opted to use subfolders rather than subdomains or cctlds. So I set up a Google Analytic Property, with one Unfiltered profile, and another 3 profiles filtered per language. With the main language English being exclude anything starting /fr/ /de/ as it resides on root. The filters seem to work fine; however after linking this to the Google Webmaster Account to be able to access Search Engine Optimization I do not seem to get any language filtered data. I was wondering if someone had any idea or possible solution to this problem. As I would expect to at least have the Landing Pages if not exactly the keywords filtered by the same criteria that the rest of the data is being filtered. I know there's also an option to create a separate webmaster tools account, however this way I still cannot filter just the English; and I cannot link it to all the separate profiles.
International SEO | | jonmifsud0 -
URL Structure - Homepage, Country and State Pages
Hello, I am creating a website (or websites if best format) that will have state-specific boating license courses for every state in the US, Canada and Australia. I would like the content to be available on the website in English, French and Spanish. I want to be the global leader in providing boat test courses. For the (1) homepage, (2) country pages, and (3) state pages, what is best SEO format I should use for:
International SEO | | Monologix
(a) URL structure
(b) "href lang" code
(c) rel canonical code
(d) will meta content with non-English pages need to also be in the non-English language of that page? Also, what server company do you recommend I host my website with? I am a non-programmer and learning SEO, so any and all help will be greatly appreciated! Thank you very much in advance!!!0 -
How to make Google consider my international subdomain relevant?
We have recently started to look deeper into international SEO. We have search engine optimized our international landing pages, title tags and meta descriptions with keywords etc. so each of the international language we support is SEO'ed for the local market. We support 12 languages, and each of them are located on a subdomain. That means if we say our site is helloworld.com, a person from Germany that lands on this site can switch to German and will then be redirected to de.helloworld.com and all content will be in German. Our problem is that we develop cloud-based software, we have a significant amount of traffic, but whenever we get media coverage or people link to us from anywhere in the world they always link to the root domain which in this case then would be helloworld.com. That means if I go to google.de and type in the exact meta description or title tag we use in German, the Google search engine can't even find us because "I assume" Google don't consider our de.helloworld.com relevant because nobody has ever linked to this site. I would appreciate very much if anyone can give me some advice on how I can address this issue. Thanks a lot! Allan
International SEO | | Todoist0 -
Different Home Sites for different Countries but same Language
We'r starting a new webshop soon and and one of our programmers came up with the following: Different Home Sites (Index Pages) for Austria and Germany. The Language is both times German but some words are different than others. The customer would like to have that. So we would have: domain.com (No Austrian or German IP Address) domain.com/at/ (User with Austrian IP Adress) domain.com/de/ (User with German IP Address) Is this SEO wise a disadvantage? How to set up the canonicals? DE & AT Page with the Canonical on the main Domain? Any advice? Thank you
International SEO | | leitpix0