How to Safely Scrape Google Results?
-
I've built a couple of small tools that I use personally, maybe 2 or 3 times per day.
Both tools scrape the top 10 results from Google and provide more details about each domain (like the SEOMoz Keyword Difficulty Tool).
Google seem to have banned my IP address for automated searches... can anyone tell me a safe way of scraping the google results? Is there a suitable API for this?
How do SEO Moz do this on such a huge scale?
-
As I doubt that the APIs have considerably improved since this blog post http://www.seomoz.org/blog/the-nasty-problem-with-scraping-results-from-the-engines, google scraping is still a big issue and necessary for our daily seo work.
Scraping savely can only work if you succeed in convincing Google that you're a "natural" user and not a scarping robot. How can you do that?
- Search with alternating IPs, from different locations using proxies from the countries where you'd like to scrape from
- don't send too many requests at once from the same source
Consider that, when requesting a URL, the browser sends various information elements to the server, containing, for example, your Operating System, browser version, referer, etc. - every element can and should be changed to virtually change your identity when executing a new search.
- change browsers, browser versions, operating system information, etc.
- take care when changing browser localization values (en-GB, en-US probably don't return the same results)
- have a good network of proxy servers ready to send the different requests with your different identities to
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Strange google search - help me mozzers
Hey Mozzers How Are You All? Hope You Are Doing Well. 🙂 I need your help. I have been having a very bad experience with Google. 😞 Do you know what is "Google Dance"? Somewhere I have heard about this. I don't have even clearance about Google Dance. BTW, the problem is that i have an adult video site about a porns tar. I was ranking at 3-6 position on a "particular keyword" on google. But about 20 days ago when i searched my "keyword term", I could not find it anywhere even on page 50 in Google search. 😄 My heartbeat was gone faster. As i was getting 25k visitors from google of that keyword. 🙂 Then i looked at my webmaster tools to check "manual spam action". There was no issue with spam or anything related to it. Then i checked my google analytics i was getting visitors from google but not from the 'particular keyword'. I was sure it was not "google hit or penalized". I could not find the exact reason why did it happen. I checked my sites everything seems to be fine. Then i did "fetch as google" and the strange thing was happened, my site pop up again in google on "that keyword". But this not the end. From that day my search keyword was getting vanish within 24 hours or more in google. And i do google fetch and it again comes to search results. 😄 I tried moz tools. no such issue. Though i cannot sort out the whole matter. What’s wrong going with google. Its an adult site, otherwise I would provide my site url and the keyword. By the way, I have attached some screenshots. I am waiting for your help. Thanks to all. 🙂 gyKIRVW
Competitive Research | | raja290 -
Can an https: impact (not provided) keywords in Google Analytics? Meaning if I make my site https: will it provide any more keywords. If not how can I get this data?
I have a site that I'd love to have more keyword data on. Can making my site an https: site give me more keyword data than leaving my site an http: site? Thanks!
Competitive Research | | mattdinbrooklyn0 -
Predicting Time For Results
Hi Mozzers. My company uses an SEO agency. They are building links from relevant sites but, since hiring them around 9 months ago our SERPs for our important search terms have remained pretty static (dropping slightly). I'm in charge of my company's marketing budget so need to make decisions about ROI when determining where best to invest my resources. It seems to me that it ought to be possible to predict (with a fair degree of accuracy) how long it's likely to take us to move up the SERPs based upon an analysis of our website, link profile and how it's changing over time vs those of our competitors. Unfortunately my SEO agency keeps dodging this question. Am I being unreasonable in expecting this? Is what I've described above even possible? If it is possible what are the factors you would consider when trying to make such a prediction? Thanks for your help David
Competitive Research | | davidoff5744440 -
How to check Google Keyword rankings?
Hey, So I recently watched the DuckDuckGo commercial about Google's results bubble. My question is how can you get the actual keyword rankings of relevant key phrases without Google taking your locations and search history into account? Would it just be a case of clearing your cache & history from the beginning of time or is there an accurate tool (other than SEOmoz ofc that I can use? Best regards, Dan
Competitive Research | | Sparkstone0 -
Quick question about country specific organic results
Do you think that if your website is from your home country. You will rank better for some keyword even when you dont have much page authority when compared to other websites having much higher page authorities from other countries.
Competitive Research | | ksbnok0 -
Trying to understand the sporadic and not relevant search results from Google?
Trying to understand the sporadic search results from Google I have been searching for answers to my situation for two
Competitive Research | | rdominey
months now with little results. I have had numerous answers and various
solutions to the issue, and I have followed through with them trying to solve
this problem but the results are the same. Beginning in late April our SERP went down drastically. I
have checked and cleaned the website of any errors and issues that I can find with
no change in the results. I have submitted a request to Google to be
reconsidered and have been told now three times that we are not being manually
penalized. The last request I asked for
any specific reason that we were seeing these very unusual SERP results and got
no answer. The best way that I can explain the issues is with a specific example. These results are similar with other pages throughout the website: www.getyourphotosoncanvas.com/ If you search for a specific page Title you get some really strange results: Search for: Gallery Wrap vs Museum Wrap Photos on Canvas The results vary from day to day but for the most part remain very similar: ·** Black and White Photos on Canvas make Amazing
Wall Art** www.getyourphotosoncanvas.com/.../black-and-white-photos-on-can... Mar 20, 2012 – ... shots to Black and White. What a great way to display your photos on canvas. ... Gallery Wrap vs. Museum Wrap · Before & Afters
That WOW! #53 | PR: ·** Sitemap - Get Your Photos on Canvas** www.getyourphotosoncanvas.com/sitemap/ Use our Sitemap to help locate exactly what you are looking for and navigate directly to that ... FREE Enhancements · MORE Enhancements · Gallery Wrap vs. #54 | PR: ·** Like Get Your Photos on Canvas on Facebook** www.getyourphotosoncanvas.com/like-us-on-facebook/ Like Us on Facebook and keep in touch with Special Offers and information about Get Your **...**FREE Enhancements · MORE Enhancements · Gallery Wrap vs. #56 | PR: ·** [Help Get Your Photos on Canvas](http://www.getyourphotosoncanvas.com/help/)** www.getyourphotosoncanvas.com/help/ Our Help Pages have information about Get Your Photos on Canvas, our Gift Cards, What our ... FREE Enhancements · MORE Enhancements · Gallery Wrap vs. #62 | PR: ·** Free Convert to Black and White for Your Photo on
Canvas Prints.** www.getyourphotosoncanvas.com/free-black-and-white-adjustments/ ... OF CHARGE. We produce the very best quality Black and White Photos on Canvas in the business. ... Wrap vs. Museum Wrap · Before & Afters That WOW! #67 | PR ·** Large Format Photographic Canvas Wall Displays - Get Your Photos ...** www.getyourphotosoncanvas.com/wall-displays/ Photo on Canvas Wall Displays or Wall Groupings can be produced in a wide variety of Layouts including ... Wrap vs. Museum Wrap · Before & Afters That WOW! #73 | PR The target page does not show up in the top 350 search results yet all of the above pages are listed in the top 100. It is important to note that most of these pages do not even contain the keywords allery Wrap vs Museum Wrap or any variation of those keywords. Yet, they appear in the displayed results? If you search just Gallery Wrap vs Museum Wrap our title page appears as #2 on Google? ·** Gallery Wrap vs Museum Wrap Photos on Canvas** www.getyourphotosoncanvas.com/gallery-wraps-vs-museum-wraps/ Gallery Wrap vs Museum Wrap Canvas, two options for your photo on canvas prints. Find out the difference between Gallery Wrap and Museum Wrap. #2 | PR Just as a point of interest, both keyword phrases score A in on page results. I would be grateful for any ideas or assistance on what might be causing these results and of course guidance on what I need to do to correct the problem. Thank You.0 -
How a site only with Frame can be at #1 google in a competitive keyword?
Guys, I need help. I have a company in Brasil and due to lack of budget, I'm doing the SEO for my self. And for two years Im reading everything that I can to do my best in my website. I worked very hard, but the #1 in the most important keyword for my company is a very bad website only with frames. What they are doing? Could anyone give help? Keyword: "acampamento"
Competitive Research | | Naghirniac
Google Brazil (www.google.com.br)
#1 website: www.acampar.net My website: www.acampamentoaguiasdaserra.com.br Thank you in advance Miguel0 -
What metrics are you using to build profiles of competitors Google Places pages?
Hi, I am attempting to put together a set of procedures to undertake when optimizing Google Places pages. Does anyone have a good list of metrics and methods for determining them in order to build profiles of competitors' Places pages? Thanks
Competitive Research | | networkelites0