What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Country and Language Specific URL Paths
Wanted to ask everyone a questions: So our company is going to be doing a website that is going to be full of videos. The url path will be country.domain.com/language/slug/content-id. We redirect the user when they go to the different country. So if you're in spain on a train to france your URL will change from es.domain.com/es/slug/content-id to fr.domain.com/es/slug/content-id. Each country can listen to each video in all languages. My question is with hreflang tags and canonicals. Aside from targeting users in a certain country via Google Search Console, how do I eliminate duplication and tell Google which I'd like to show up via which country. In spain I would like es.domain.com/es/slug/content-id to show in Google and would have hreflang tags on each of the es.domain pages but what about fr.domain.com/es/slug/content-id since it would show the same content? I can't canonical to one of them since I need them to show in their respective country. How do I show the difference in language and country without showing duplication?
International SEO | | mattdinbrooklyn0 -
How to avoid duplication across multiple country domains
Here's the scenario: I have a client currently running one Shopify site (AU) They want to launch three more country domains (US, UK and EU) They want each to be a standalone site, primarily so the customers can purchase in their local currency, which is not possible from a single Shopify site The inventory is all from the same source The product desscriptions will all be the same as well Question: How do we avoid content duplication (ie. how will canonical tags work in this scenario)?
International SEO | | muzzmoz0 -
Recent Google Link Scheme Updated ? What's Your Reaction against Link Building, Link Exchanging ?
Many Bloggers and Webmasters are upset over this !
International SEO | | Esaky
Recent Google Link Scheme Updated ? What's Your Reaction against Link Building, Link Exchanging ? https://support.google.com/webmasters/answer/66356?hl=en What will you Do, if we are good at traffic to our blog and advertiser link will be no-follow - will they accept it ! and guest post also. They need a do-follow link back to their blog or website they hired for !0 -
Poor Google.co.uk ranking for a UK based .net, but great Google.com
I run an extremely popular news & community website at http://www.onedirection.net, but we're having a few ranking issues in Google.co.uk. The site gets most of its traffic from the USA which isnt a bad thing - but for our key term "one direction", we currently don't rank at all on Google.co.uk. The site is located on a server based in Manchester, UK, and we used to rank very well earlier this year - fluttering about in position 5-7 most of the time. However earlier this year, around July, we started to fall down to page 2 or 3, and at the start of this month we don't rank at all for "one direction" on Google.co.uk. On Google.com however we're very strong, always on page one. We're definitely indexed on .co.uk, just not for main search term - which I find a bit frustrating. All the content on our site is unique, and we write 2-4 stories every day. We have an active forum too, so a lot of our content is user-generated. We've never had any "unnatural link building" messages in Webmaster Tools, and our link profile looks fine to me. Do we just need more .co.uk links, or are we being penalised for something? (I can't imagine what though). It certainly seems that way though. Another site, "www.onedirection.co.uk" which is never updated and has a blatant ad for something completely unrelated on its homepage, ranks above us at the moment- which I find quite frankly appalling as our site is pretty much regarded as the worlds most popular One Direction news and fan site. We've spent the last few months improving the page-load times of our site, and we've reduced any unneccesary internal linking on the site. Approx 2 months ago we launched a new forum on the site, 301'ing all the old forum links to the new one, so that could have had an impact on rankings - but we'd expect to see an impact on Google.com as well if this was an issue. We definitely feel that we should be ranking higher on Google.co.uk. Does anyone have any ideas what the iproblems could be? Cheers, Chris.
International SEO | | PixelKicks0 -
How to optimise a site for 2 countries
Hi there - Any help with the below much appreciated I am helping an Australian company, producing packaging products for businesses. Their site is hosted in Australia and their offices are in Australia. They have asked me to take care of both on-page and off-page SEO so that they rank for keywords related to their products - e.g. 'cardboard boxes'. This should be fairly straightforward for Australian based (.com.au) searchers, but they also supply their products to South Africa, and so want their results to show up also for South African based (.co.za) searchers. Also consider: it is not typical for searchers for these products to use geomodifiers in their search terms there is no unique content for the South African market versus the Australian... the product information is essentially identical. What should we do to ensure their results show up equally for those in South Africa as well as Australia? I am considering building a completely separate site, hosted in South Africa and specifically for the S.A market, but will the duplicate content effect be an issue? Also, this would essentially mean double the SEO effort, is there no way I could achieve our goals more efficiently? many thanks to any help
International SEO | | dnaynay0 -
Http://us.burberry.com/: Big traffic change for top URL (error 593f1ceb2d67)
Please forgive duplicating this question on the SEOMoz & Webmaster Tools forum but I'm hoping to hit both audiences with this question... A few days ago I noticed that our US homepage (us.burberry.com) had dropped from PR5 to PR0, and the page has been deindexed by Google. After checking Webmaster Tools I also received the following message: http://us.burberry.com/: Big traffic change for top URL April 2, 2012Search results clicks for http://us.burberry.com/ have decreased significantly.Message ID: 593f1ceb2d67.We're not doing any link building at all (we've enough on-site issues to deal with). The only changes I have made are adding Google Analytics to the website, uploading sitemaps via Webmaster Tools (it's not linked to from robots.txt yet), and setting the burberry.com and www.burberry.com geo-location settings to 'unlisted' (we want uk.burberry.com appearing in the UK results, us.burberry.com appearing in the US results etc rather than www.burberry.com).I've reversed the geo-location settings but I doubt this would have caused this. We've duplicate copies of our homepage (such as us.burberry.com/store//) from typos in inbound links (and bad programming that allows them to work rather than 404'ing) but I don't think any of this is new. What I don't understand is (a) why this is happening now and (b) why is this just affecting our US homepage? We've ~40 different duplicates of the homepage (us, uk, ca, pt, ro, sk etc etc) so why is the US site being affected and not the others? Does anyone know if this is due to an algorithm change by Google or something else all together? Background:Our website www.burberry.com has 46 subdomains such as uk.burberry.com, ca.burberry.com and us.burberry.com. There is a lot of duplicate content on each subdomain (including basic things like tracking parameters in URLs) and across subdomains (uk.burberry.com/store & us.burberry.com/store are exactly the same), there's very little text on the site (its nearly all images), as well as poor redirects, inaccessible content (AJAX/Flash) and a whole host of basic SEO things that aren't being done correctly. I've joined the company in the last few months and have started addressing these issues but I've got a LOT of work to do yet.One thing that we have in our favour is a link profile that is as clean and natural as they come - there was only ever one link building campaign performed (which was before my time) and I had all of those links removed as soon as I joined the company.Any help would be greatly appreciated! Thanks for your timeDean RoweEdit: us.burberry.com 301 redirects to us.burberry.com/store/ as explained on the webmaster tools forum, but I don't believe this is the cause as its the same across all subdomains.
International SEO | | FashionLux0 -
What is the best way to make country specific IP redirect for only product pricng pages?
My website has 3 services and its price will be different for US/EU/Developed world and Asian/African countries.Apart from pricing page, all other things remain same. I want to use IP based redirect .I heard this thing is called cloaking and used by black-hat guys. What kind of instructions should I give to my web developer to look best to Google/Search bots and correctly show visitors the intended prices.Is there any caution to be taken care of. Thanks for your time
International SEO | | RyanSat0 -
How to improve SERP rankings in other countries?
We have a .com website in English targeting global visitors. We have good rankings on Google USA for our targeted keywords But we do not rank well in APAC e.g. Sigapore and Australia. What could we do to get ranked in these countries? We received a recommendation to sign up for .sg and .au domains and replicate our Website for Singapore and Australia. But wouldn’t it contribute to duplicate content Issue? We are in software industry and do not have locale specific content. Please advise.
International SEO | | Amjath0
