What IP Address does Googlebot use to read your site when coming from an external backlink?
-
Hi All,
I'm trying to find more information on what IP address Googlebot would use when arriving to crawl your site from an external backlink.
I'm under the impression Googlebot uses international signals to determine the best IP address to use when crawling (US / non-US) and then carries on with that IP when it arrives to your website?
E.g. - Googlebot finds www.example.co.uk. Due to the ccTLD, it decides to crawl the site with a UK IP address rather than a US one. As it crawls this UK site, it finds a subdirectory backlink to your website and continues to crawl your website with the aforementioned UK IP address.
Is this a correct assumption, or does Googlebot look at altering the IP address as it enters a backlink / new domain?
Also, are ccTLDs the main signals to determine the possibility of Google switching to an international IP address to crawl, rather than the standard US one? Am I right in saying that hreflang tags don't apply here at all, as their purpose is to be used in SERPS and helping Google to determine which page to serve to users based on their IP etc.
If anyone has any insight this would be great.
-
There's a few things you need to marry up if you want to do this. You need the referring page or domain / hostname (to validate that the session came from a backlink you know about). Once you filter the data down like that, you just need to filter by user-agent ("googlebot" - or any user-agent string which contains "googlebot"). Then you just want to look at the IP address field in the tabular data and you have your answers!
Here's the problem, most IP-level data is contained within basic server-side analysis packages (like AWStats which is installed on most sites, within the cPanel) or alternatively you can go to the log files for much of the same data. Most referrer-level data (stuff that deals with attribution) is contained within Analytics suites like Adobe Omniture or Google Analytics.
In GA, you can't usually get to 'individual' IP-level data. There used to be a URL hack to force it to render, but it was killed off (and many people who used it were banned by Google). The reason for that is, Google don't want too much PID (Personally Identifiable Data) harvested by their tool. It creates too many legal issues for Google (and also, whomever is leveraging that data for potentially nefarious marketing purposes)
Since you won't get enough IP-level data from GA, you're going to have to go to log files and log analysis tools instead. Hopefully they will contain at least some referral level data... The issue is, getting all the pieces you want to align in a legally compliant way
Obviously you have your reasons for looking. I'd check if you can find anything on your CPanel in AWStats (if that's installed) or get the log files and analyse them with something like Screaming Frog Log File Analyser
I can't promise this will return the data you want, but it's probably your only hope
-
Hi,
First of all "Google crawls from many IPs and they have confirmed that they do periodically add new ones. And there are also various Googlebot useragents, not just the regular one. This is why Google doesn't publish a list of all the IPs, because there are so many of them and they can change" .
You can see full conversation here @ https://productforums.google.com/forum/#!msg/webmasters/4fKthSy7oFQ/GgslLXJnDQAJ
Second Today Google says "IP Addresses Don't Matter For Backlinks & Search Rankings"
https://www.seroundtable.com/google-ip-addresses-backlinks-rankings-26561.html
Hope this helps
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 redirecting a site that currently links to the target site
I have a personal blog that has a good amount of back links pointing at it from high quality relevant authoritative sites in my niche. I also run a company in the same niche. I link to a page on the company site from the personal blog article that has bunch of relevant links pointing at it (as it's highly relevant to the content on the personal blog). Overview: Relevant personal blog post has a bunch of relevant external links pointing at it (completely organic). Relevant personal blog post then links (externally) to relevant company site page and is helping that page rank. Question: If I do the work to 301 the personal blog to the company site, and then link internally from the blog page to the other relevant company page, will this kill that back link or will the internal link help as much as the current external link does currently? **For clarity: ** External sites => External blog => External link to company page VS External sites => External blog 301 => Blog page (now on company blog) => Internal link to target page I would love to hear from anyone that has performed this in the past 🙂
Intermediate & Advanced SEO | | Keyword_NotProvided0 -
Site map creator
I have a large website with about 1300 pages. I can't find a good sitemap creator that will crawl the whole site and spit out the xml file. Any ideas or suggestions for good services? Also, a site this large, should I consider mutiple site maps?
Intermediate & Advanced SEO | | dwebb0070 -
Bit.ly backlinks
Hi all, what experience do you have with Bit.ly links? Can I use it for backlinking management?
Intermediate & Advanced SEO | | Tormar3 -
How is my 301 redirected site stealing rankings from the main site?
Hello, I have a site, drhobelt.com, that 301 redirects to the main site, drhonow.com. Not only is drhobelt.com still indexed, but it recently stole rankings from drhonow.com for "decompression belt" related terms. What could be causing this? How do I reclaim the rankings for drhonow.com? Thanks for reading!!
Intermediate & Advanced SEO | | DA20130 -
Potential problems with my site
Dear Mozzers I hope you can help me with the following problems: My site is up and running for a year now and may be there has been problem with the homepage, because it ranks on first page for a competitive keyword on Google.com and Google.com.au only, however in other countries it just shows up as internal page and does not rank well. google.com/Google.com.au: homepage ranks top 10 (example.com)
Intermediate & Advanced SEO | | SteveTran2013
Other countries (.co.uk, .ca..ect) an internal page shows up, example.com/internalpage.html - shows on page 3-4. I can not find the homepage of example.com anywhere around top 1000. Can you please tell me what are the potential problems. Thank you very much. BR/Tran0 -
How to make an AJAX site crawlable when PushState and #! can't be used?
Dear Mozzers, Does anyone know a solution to make an AJAX site crawlable if: 1. You can't make use of #! (with HTML snapshots) due to tracking in Analytics 2. PushState can't be implemented Could it be a solution to create two versions of each page (one without #!, so campaigns can be tracked in Analytics & one with #! which will be presented to Google)? Or is there another magical solution that works as well? Any input or advice is highly appreciated! Kind regards, Peter
Intermediate & Advanced SEO | | ConversionMob0 -
IP address being indexed by Google in addition to canonical domain.
Our site's IP address is being indexed in addition to the canonical www.example.com domain. As soon as it was flagged a 301 was implemented in the .htaccess file to redirect the IP address to the canonical. Does this usually occur? Is it detrimental to SEO? In my time in SEO I've never heard of this being an issue, or being part of a list of things to be checked. It sounds more like a server that wasn't configured correctly when hosting was set up? It didn't seem to be affecting the site at all, but is it more common and I've just never heard of it? 😛 Should it be something I'm usually looking for in future? Responses are greatly appreciated!
Intermediate & Advanced SEO | | mikeimrie0 -
Can your site be penalised by backlinks?
Hi, I just wanted to get some clarification on whether Google would penalize your site if you had many links coming from a questionable site. We've been struggling with rankings for years even though we have one of the oldest sites in the industry with a good link profile and the site is well optimized. I was looking through webmaster tools and noticed that one website links to us over 100,000 times, all to the home page. The site is www.vietnamfuntravel.com. When I looked at the site it seems that they operate a massive links exchange, I'm not sure what the history is and why they link to us so much though. Is there any chance that this could impact us negatively? if it is then what would be the best way to deal with the situation? I could ask them to take the links down but can't guarantee they would do it quickly (if at all). Would blocking their domain from our htaccess file have the desired effect?
Intermediate & Advanced SEO | | Maximise0