How to Safely Scrape Google Results?
-
I've built a couple of small tools that I use personally, maybe 2 or 3 times per day.
Both tools scrape the top 10 results from Google and provide more details about each domain (like the SEOMoz Keyword Difficulty Tool).
Google seem to have banned my IP address for automated searches... can anyone tell me a safe way of scraping the google results? Is there a suitable API for this?
How do SEO Moz do this on such a huge scale?
-
As I doubt that the APIs have considerably improved since this blog post http://www.seomoz.org/blog/the-nasty-problem-with-scraping-results-from-the-engines, google scraping is still a big issue and necessary for our daily seo work.
Scraping savely can only work if you succeed in convincing Google that you're a "natural" user and not a scarping robot. How can you do that?
- Search with alternating IPs, from different locations using proxies from the countries where you'd like to scrape from
- don't send too many requests at once from the same source
Consider that, when requesting a URL, the browser sends various information elements to the server, containing, for example, your Operating System, browser version, referer, etc. - every element can and should be changed to virtually change your identity when executing a new search.
- change browsers, browser versions, operating system information, etc.
- take care when changing browser localization values (en-GB, en-US probably don't return the same results)
- have a good network of proxy servers ready to send the different requests with your different identities to
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What to do about one site dominating search results? (multiple pages ranking)?
Anybody have thoughts on dealing with search results where the same site gets listed multiple times? "weebly vs wix" is one example (same site #1-3, repetitive articles, not crazy high authority), but I see this now and then. I know Google likes variety, so it's weird for me to see results like this dominating search results. Thoughts? What gets these sites to take over the top rankings for a specific term? Any way to rise up in this situation, outside of the usual? Any tips on duplicating this kind of success?
Competitive Research | | davidwaring0 -
I Got A Scraper Delisted From Google ...
I have an electronics niche news website. A scraper who had an online store selling products in my niche copied every one of my articles and posted them on his site under the heading "News" ... generally within 1/2 hour of me posting them on my site. His site was even showing up in the rankings before mine. I filed a copyright infringement claim with Google two weeks ago via their online form explaining what he was doing. Today, I received an email from Google saying that they have reviewed his site and have delisted it from the search engine. I just checked, and he is GONE ... completely delisted, no trace. My site traffic has also jumped at least 25% today. It pays to complain! Just sharing 😉
Competitive Research | | Humanovation3 -
Does anyone use the Google Custom Search API?
What has your experience been like with the API? Do you prefer a rank tracking tool? If so, which one? API: https://developers.google.com/custom-search/v1/overview
Competitive Research | | CIEEwebTeam0 -
How do I know top pages (first on Google) for one website?
Hi! I'm SEOMOZ PRO user and I want to know which pages of other website are ranking on top on Google, Yahoo, Bing, ecc... with relative position (rank). Is there a features of SEOMOZ for it? Regards
Competitive Research | | jadlib0 -
My site is ranked in the top 5 for my keywords, but howcome I'm low in an organic search results for my key words?
Are the other factors such as page rank, Alexa rating and mozRank used to determine where I will show up in search results, over Goggle's key word rank for my key words?
Competitive Research | | allstatetransmission0 -
When providing search results for SEO purposes to you use the exact results in Google Adwords
Hi Mozzers Just quick question When an SEO company are supplying their testimonials for example
Competitive Research | | mcliddy
Keyword Search Term has 33,000 visits a month
Keywords is in position 1 but the search volume they are showing is broad, i was always brought up to do research on exact results unless im using the reserch for a PPC campagin? Has anyone got any ideas?? should it be braod im looking at or exact?? Many Thanks Matt0 -
Why do i not receive google traffic?
İ have published over 3000 unique articles to pr3 drupal site over the past 3 months, yet only get about 20-30 visitors a day from google to my new 3000 articles. i have spent over 10 000usd for those articles, all range between 400-800 words and all pass copyscape. 90 percent of the articles are indexed and site pr3 site. the site is alltopics.com why do i not receive traffic?
Competitive Research | | rxesiv0 -
My client has shown me a similar site, though not a competitor. He wants to know what sites they are linked from that give them such a good Google rank for certain kewords. Can SEOMoz tell me this?
When using google.com.au and searching for "travel to france", www.frenchtravel.com.au is the 3rd organic result. (the 1st two are not travel businesses, they are non profit travel guides) My client, who runs www.visituk.com.au, an Australian site that organises tours of the UK, said "so we just need to add these sort of words to the site?" I said, yes, but it doesn't end there. The real task is to have a link to your site on other sites surrounded with the words "travel" and "UK". He asked if he could see a list of the sites the french site was being referred by relevant to the search phrase. Is there an SEOmoz tool for this? Or is there another way I can generate that list? Thanks Simon
Competitive Research | | electrik0