How to Safely Scrape Google Results?

[cmaddison]

I've built a couple of small tools that I use personally, maybe 2 or 3 times per day.

Both tools scrape the top 10 results from Google and provide more details about each domain (like the SEOMoz Keyword Difficulty Tool).

Google seem to have banned my IP address for automated searches... can anyone tell me a safe way of scraping the google results? Is there a suitable API for this?

How do SEO Moz do this on such a huge scale?

[gmellak]

As I doubt that the APIs have considerably improved since this blog post http://www.seomoz.org/blog/the-nasty-problem-with-scraping-results-from-the-engines, google scraping is still a big issue and necessary for our daily seo work.

Scraping savely can only work if you succeed in convincing Google that you're a "natural" user and not a scarping robot. How can you do that?

• Search with alternating IPs, from different locations using proxies from the countries where you'd like to scrape from
• don't send too many requests at once from the same source

Consider that, when requesting a URL, the browser sends various information elements to the server, containing, for example, your Operating System, browser version, referer, etc. - every element can and should be changed to virtually change your identity when executing a new search.

• change browsers, browser versions, operating system information, etc.
• take care when changing browser localization values (en-GB, en-US probably don't return the same results)
• have a good network of proxy servers ready to send the different requests with your different identities to