What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How should I handle hreflang tags if it's the same language in all targeting countries?
My company is creating an international version of our site at international.example.com. We are located in the US with our main site at www.example.com targeting US & Canada but offering slightly different products elsewhere internationally. Ideally, we would have hreflang tags for different versions in different languages, however, it's going to be an almost duplicate site besides a few different SKUs. All language and content on the site is going to be in English. Again, the only content changing is slightly different SKUs, they are almost identical sites. The subdomain is our only option right now. Should we implement hreflang tags even if both languages are English and only some of the content is different? Or will having just canonicals be fine? How should we handle this? Would it make sense to use hreflang this way and include it on both versions? I believe this would be signaling for US & Canda visitors to visit our main site and all other users go to the international site. Am I thinking this correctly or should we be doing this a different way?
International SEO | | tcope250 -
How well does Google's "Locale-aware crawling by Googlebot" work?
Hello, In January of this year Google introduced "Locale-aware crawling by Googlebot." https://support.google.com/webmasters/answer/6144055?hl=e Google uses different crawl settings for sites that cannot have separate URLs for each locale. ......... This is basically for sites that dynamically render contend on the same URL depending on the locale and language (IP) of the visitor. If e.g. a visitor was coming from France, the targeted page would load in french. If a visitor was coming from the US the same page would load in English on the same URL. Does anyone have any experience with this setup and how well it works? How well do the different versions of a page get indexed, and how well do those pages rank? In the example above, does the french content get indexed correctly? Many thanks!
International SEO | | Veva0 -
Recent Google Link Scheme Updated ? What's Your Reaction against Link Building, Link Exchanging ?
Many Bloggers and Webmasters are upset over this !
International SEO | | Esaky
Recent Google Link Scheme Updated ? What's Your Reaction against Link Building, Link Exchanging ? https://support.google.com/webmasters/answer/66356?hl=en What will you Do, if we are good at traffic to our blog and advertiser link will be no-follow - will they accept it ! and guest post also. They need a do-follow link back to their blog or website they hired for !0 -
Ranking well internationally, usage of hreflang, duplicate country content
I'm trying to wrap my head around various options when it comes to international SEO, specifically how to rank well in countries that share a language, and the risk of duplicate content in these cases. We have a chance to start from scratch because we're switching to a new e-commerce platform, and we were looking into using hreflang. Let's assume an example of a .com webshop that targets both Austria and Germany. One option is to include both language and region in the URL, and mark these as such using hreflang: webshop.com/de-de/german-language-content (with hreflang de-de)
International SEO | | DocdataCommerce
webshop.com/de-at/german-language-content (with hreflang de-at) Another option would be to only include the language in the URL, not the region, and let Google figure out the rest: webshop.com/de/german-language-content (with hreflang de) Which would be better? The risk of inserting a country, of course, is that you're introducing duplicate content, especially since for webshops there are usually only minor differences in content (pricing, currency, a word here and there). If hreflang is an effective means to make sure that visitors from each country get the correct URL from the search engines, I don't see any reason not to use this way. But if search engines get it wrong, users will end up in the wrong page and will have to switch country, which could result in conversion loss. Also, if you only use language in the URL, is it useful at all to use hreflang? Aren't engines perfectly able to recognize language already? I don't mention ccTLDs here because most of the time we're required to use a .com domain owned by our customer. But if we did, would that be much better? And would it still be useful to use hreflang then? webshop.de/german-language-content (with hreflang de-de)
webshop.at/german-language-content (with hreflang de-at) Michel Hendriks
Docdata Commerce0 -
Hotel Multi country targeting + Google Local + TLDs => "MesSEO"
Hi guys, I own a guesthouse which is facing a messy structural problem in its own web presence: Portuguese: www.residencia-aeminium-coimbra.com.pt English: www.residencia-aeminium-coimbra.com Spanish: www.residencia-aeminium-coimbra.com/espanol Looking for success in the long term, a few years ago, we decided to host 3 TLDs: 1 for global international english, 1 for local portuguese and 1 for the main foreign market Spain (we already redirected it to a subfolder in the .com in the meanwhile). We tried to promote each one of these in their targeted markets - mainly the .com got back-links links and authority. With time, google local appeared and changed the face of google accommodation rankings. Google local are now the top results for most searches, even outside the country, and I came to the conclusion that the TLD assets were now a problem. For instance, I can only add 1 domain to google local..what language should this be? It is the same for most social media presence..How should international markets be targeted in a local page? Since TLDs do not seem to be an advantage right now, I am thinking we should be moving to a single domain and use a folder structure so we can use it everywhere. Questions: 1. In terms of structure (TLD or Folder) and multi-lingual targeting what is the current best practice for hotels that show in local results but promote internationally? 2. What language should I point google local results to? And our Facebook page, etc? 3. If I move things around to a folder structure, what domain should I use? the .com is in english and has the most authority and links according to opensiteexplorer the .com.pt is in portuguese, our local language and our main market (but only 35% share) should I create a new domain ".pt"? 4. I don't think that geo-targeting the languages is worth it in this case, what do you think? Kind Regards
International SEO | | retsimister
Ricardo Madeira
Residencia Aeminium Coimbra0 -
Does publishing your physical company address increase your Google Trust?
The question is in the title: Does publishing your physical company address increase your Google Trust? I.e. on the contact page of the website, providing a physical address. The address would match the WHOIS entry for the domain, and allow customers to contact you via post.
International SEO | | Peter2640 -
Link Building in russian speaking countries
Hello, We are an international company, based mostly in Romania and Hungary, but we also launched some websites in Ukraine. The interesting thing in Ukraine that we are facing heavily, is how they are doing link building. Basically the only link building that they do is buying links through a platform: sape.ru (http://www.mainlinkads.com/ this is English version). Local SEO consultants also advice to buy links. It wasn't so much a problem, but when you have a good site with good content, but have competitors that are buying links and then overcome our good keyword positions, we started to questions whether to start buying links ourselves. What is your opinion on this? Is it possible that Google will penalize a big bunch of sites, now in sape.ru there are: 487 069 sites? Can we adopt the same strategy as our competitors as long as it is working? Thanks, Irina
International SEO | | InformMedia0 -
Different country, same language
I have read the blog posts by Rand and other community members at YouMoz but i still have a question on trageting and domains / sub-directories usage. Suppose, my business is located in France but my prospects are in US and UK as well. The issue is, they are not English speakers but French. If i use ccTLD, i don't think it will rank well in US and UK. gTLD will not be a good option for prospects in France. What should i do? Regards, Shailendra
International SEO | | IM_Learner1