How to Safely Scrape Google Results?
-
I've built a couple of small tools that I use personally, maybe 2 or 3 times per day.
Both tools scrape the top 10 results from Google and provide more details about each domain (like the SEOMoz Keyword Difficulty Tool).
Google seem to have banned my IP address for automated searches... can anyone tell me a safe way of scraping the google results? Is there a suitable API for this?
How do SEO Moz do this on such a huge scale?
-
As I doubt that the APIs have considerably improved since this blog post http://www.seomoz.org/blog/the-nasty-problem-with-scraping-results-from-the-engines, google scraping is still a big issue and necessary for our daily seo work.
Scraping savely can only work if you succeed in convincing Google that you're a "natural" user and not a scarping robot. How can you do that?
- Search with alternating IPs, from different locations using proxies from the countries where you'd like to scrape from
- don't send too many requests at once from the same source
Consider that, when requesting a URL, the browser sends various information elements to the server, containing, for example, your Operating System, browser version, referer, etc. - every element can and should be changed to virtually change your identity when executing a new search.
- change browsers, browser versions, operating system information, etc.
- take care when changing browser localization values (en-GB, en-US probably don't return the same results)
- have a good network of proxy servers ready to send the different requests with your different identities to
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What basics should we be looking at in Google Analytics?
We have GA set up on our site for a while now. However, we do not look at the data. We did today and it's quite overwhelming. Is there anything in specific we should start with? A top 3 items we should start to monitor each week and try to improve? Bonus Question: Is there a way to use analytics to find out who are competitors are?
Competitive Research | | IcarusSEO0 -
Strange google search - help me mozzers
Hey Mozzers How Are You All? Hope You Are Doing Well. 🙂 I need your help. I have been having a very bad experience with Google. 😞 Do you know what is "Google Dance"? Somewhere I have heard about this. I don't have even clearance about Google Dance. BTW, the problem is that i have an adult video site about a porns tar. I was ranking at 3-6 position on a "particular keyword" on google. But about 20 days ago when i searched my "keyword term", I could not find it anywhere even on page 50 in Google search. 😄 My heartbeat was gone faster. As i was getting 25k visitors from google of that keyword. 🙂 Then i looked at my webmaster tools to check "manual spam action". There was no issue with spam or anything related to it. Then i checked my google analytics i was getting visitors from google but not from the 'particular keyword'. I was sure it was not "google hit or penalized". I could not find the exact reason why did it happen. I checked my sites everything seems to be fine. Then i did "fetch as google" and the strange thing was happened, my site pop up again in google on "that keyword". But this not the end. From that day my search keyword was getting vanish within 24 hours or more in google. And i do google fetch and it again comes to search results. 😄 I tried moz tools. no such issue. Though i cannot sort out the whole matter. What’s wrong going with google. Its an adult site, otherwise I would provide my site url and the keyword. By the way, I have attached some screenshots. I am waiting for your help. Thanks to all. 🙂 gyKIRVW
Competitive Research | | raja290 -
Google Keyword Tool Alternatives
Hey All, So ever since Google shut down the Keyword Tool we have used the keyword planner and a few other tools only to find they aren't quite the same, I was wondering what everyone is using now to get there keyword research and average traffic estimates ? We previous used keyword tool for phrase match and now not much offers phrase match or provides us with what seems as accurate results. We have tried Word Tracker after a numerous amount of research to find a alternative to the keyword tool, and found that the results are very different from the Google Keyword Planner, 1 noticeable result would be of "Web Design Melbourne", in Word Tracker it showed the average search of 9,900 (per month or year I'm not sure) and we compared that to the Google Keyword Planner and it showed 3,000. I have looked over a number of tools and found they weren't quite what we are looking for (Word Tracker, SEMRush, Google Keyword Planner) so I have turned to the MOZ Community. So my first question for everyone would be: A) what do you use for Keyword research and average search estimates now (any good tools with phrase match)? B) If you have used Word Tracker have you found accurate results in your keyword research? If you have found any tools that have proved beneficial and useful please let me know 🙂 (paid or free) Thank you
Competitive Research | | KBB_Digital
Jake Crone0 -
Google serps for split keyword searches
A client has come to me with an issue with the marketing of their website. it is mainly focused around the search of their name. the site is an Magento GO hosted e-commerce based site through Magento. it requires a good amount of optimisation and its in a very competitive market - which has high volume of searches. When entering the name of the business it doesn't show on the first page of Google but it does on Bing or Yahoo. The site name is lovehugz and is UK based The issue is when searched for by name lovehugz it appears on the second page, however if the search term is separated love hugz it appears first on the first page. The site is poorly optmised at present with the page title limited to the name lovehugz. Does anybody know of reasons why Google might be behaving in this manner and what steps should be taken to overcome. Optimising for the name is not the only aim here, however it something that should be in place for people searching by name, There is a lot of work to be done to optimise the site for its essential relevant keywords to stand a chance in its market. Any ideas of help would be greatly appreciated thanks
Competitive Research | | Bristolweb0 -
Trying to understand the sporadic and not relevant search results from Google?
Trying to understand the sporadic search results from Google I have been searching for answers to my situation for two
Competitive Research | | rdominey
months now with little results. I have had numerous answers and various
solutions to the issue, and I have followed through with them trying to solve
this problem but the results are the same. Beginning in late April our SERP went down drastically. I
have checked and cleaned the website of any errors and issues that I can find with
no change in the results. I have submitted a request to Google to be
reconsidered and have been told now three times that we are not being manually
penalized. The last request I asked for
any specific reason that we were seeing these very unusual SERP results and got
no answer. The best way that I can explain the issues is with a specific example. These results are similar with other pages throughout the website: www.getyourphotosoncanvas.com/ If you search for a specific page Title you get some really strange results: Search for: Gallery Wrap vs Museum Wrap Photos on Canvas The results vary from day to day but for the most part remain very similar: ·** Black and White Photos on Canvas make Amazing
Wall Art** www.getyourphotosoncanvas.com/.../black-and-white-photos-on-can... Mar 20, 2012 – ... shots to Black and White. What a great way to display your photos on canvas. ... Gallery Wrap vs. Museum Wrap · Before & Afters
That WOW! #53 | PR: ·** Sitemap - Get Your Photos on Canvas** www.getyourphotosoncanvas.com/sitemap/ Use our Sitemap to help locate exactly what you are looking for and navigate directly to that ... FREE Enhancements · MORE Enhancements · Gallery Wrap vs. #54 | PR: ·** Like Get Your Photos on Canvas on Facebook** www.getyourphotosoncanvas.com/like-us-on-facebook/ Like Us on Facebook and keep in touch with Special Offers and information about Get Your **...**FREE Enhancements · MORE Enhancements · Gallery Wrap vs. #56 | PR: ·** [Help Get Your Photos on Canvas](http://www.getyourphotosoncanvas.com/help/)** www.getyourphotosoncanvas.com/help/ Our Help Pages have information about Get Your Photos on Canvas, our Gift Cards, What our ... FREE Enhancements · MORE Enhancements · Gallery Wrap vs. #62 | PR: ·** Free Convert to Black and White for Your Photo on
Canvas Prints.** www.getyourphotosoncanvas.com/free-black-and-white-adjustments/ ... OF CHARGE. We produce the very best quality Black and White Photos on Canvas in the business. ... Wrap vs. Museum Wrap · Before & Afters That WOW! #67 | PR ·** Large Format Photographic Canvas Wall Displays - Get Your Photos ...** www.getyourphotosoncanvas.com/wall-displays/ Photo on Canvas Wall Displays or Wall Groupings can be produced in a wide variety of Layouts including ... Wrap vs. Museum Wrap · Before & Afters That WOW! #73 | PR The target page does not show up in the top 350 search results yet all of the above pages are listed in the top 100. It is important to note that most of these pages do not even contain the keywords allery Wrap vs Museum Wrap or any variation of those keywords. Yet, they appear in the displayed results? If you search just Gallery Wrap vs Museum Wrap our title page appears as #2 on Google? ·** Gallery Wrap vs Museum Wrap Photos on Canvas** www.getyourphotosoncanvas.com/gallery-wraps-vs-museum-wraps/ Gallery Wrap vs Museum Wrap Canvas, two options for your photo on canvas prints. Find out the difference between Gallery Wrap and Museum Wrap. #2 | PR Just as a point of interest, both keyword phrases score A in on page results. I would be grateful for any ideas or assistance on what might be causing these results and of course guidance on what I need to do to correct the problem. Thank You.0 -
What is the best way to know your accurate ranking on google? what analytics are most trustworthy?
with seomoz, i rank on google in one spot. With market samurai, I rank in another spot, With KPMRS (or whatever) i rank another place. But then when i just type my keywords into google, it puts me in a whole different world... am confused... and the client i am trying to help rank higher is confused. Thanks for any help you can offer be blessed bd
Competitive Research | | creativeguy0 -
Why do i not receive google traffic?
İ have published over 3000 unique articles to pr3 drupal site over the past 3 months, yet only get about 20-30 visitors a day from google to my new 3000 articles. i have spent over 10 000usd for those articles, all range between 400-800 words and all pass copyscape. 90 percent of the articles are indexed and site pr3 site. the site is alltopics.com why do i not receive traffic?
Competitive Research | | rxesiv0 -
Can i have chance to rank higher than official website in google local domains ?
Q : can i have chance to rank higher than official website in google local domains ? for example : rank higher than microsoft,kaspersky,nokia etc... in google italy or google germany or any other local domain for google
Competitive Research | | activeacts0