Thoughts on scraping SERPs and APIs
-
I'm perplexed about a couple of things.
Google says that scraping keyword rankings is against their policy from what I've read. Bummer. We comprise a lot of reports and manual finding and entry was a pain. Enter Moz! We still manually check and compare, but it's nice having that tool. I'm confused now though about practices and getting SERPs in an automated way. Here are my questions
- Is it against policy to get SERPs from an automated method? If that is the case, isn't Moz breaking this policy with it's awesome keyword tracker?
- If it's not, and we wanted to grab that kind of data, how would we do it? Right now, Moz's API doesn't offer this data. I thought Raven Tools at one point offered this, but they don't now from what I've read. Are there any APIs out there that we can grab this data and do what we want with it? (let's day build our own dashboard)?
Thanks for any clarification and input!
-
You're much too kind
Happy New Year!
-
Hi Rand. Anyone ever tell you you're awesome?
I just did. Thanks bud
-
Hi Boogily,
A) In terms of ToS, I'm not 100% clear. I know that Google has been strict about automated rank checking programs installed on a user's local machine, and they've enforced that a bit through technical and guidelines means. However, they've been unclear about companies like Moz, Searchmetrics, GetStat, AuthorityLabs, Conductor, Brightedge, Yield, RankAbove, and hundreds of others which provide this data in their software. I suspect they don't love it, but tolerate it and haven't taken action for over a decade now.
B) On the API front, check with GetStat and AuthorityLabs. I believe they both have rank data offerings from an API standpoint. Moz doesn't offer that and probably won't in the future, either.
Cheers!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the difference between my pages' keyword rank and keyword-specific SERP?
I've been tasked with identifying high volume keywords for products our company sells and optimizing our copy and listings to rank higher on Google. The Keyword Explorer and On-Page Grader have been fantastic for this, but as I write my reports, i'm not sure whether SERP or Moz's keyword page rank would be more accurate for reporting results and movement. For instance, one of our product pages ranks #12 for the highest (relevant) volume keyword, but when looking at the SERP analysis for that keyword, our product page actually shows up at #7 down the list. I've tried naturally searching in Google in case it was an issue of private results based on my user data but it seems to give the same result. Am I conflating two different tools? Or is the Moz rank just an estimate?
Moz Pro | | BJake0 -
Can I calculate "Keyword Difficulty" metric using Mozscape API data?
We already have a web application that pulls certain metrics about websites using the Mozscape API, but we are wanting to extend the usefulness of this application to enable users of the app to pull "Keyword Difficulty" metrics in bulk, instead of one at a time (or 5 at a time). I wouldn't mind the 5 at a time limitation if we could just automate the API calls and let the tool pull data for 50 or so keywords without user-interaction. I know that it's a "formula", but I don't know what SEOMoz uses for it's formula. Has anyone figured out a way to calculate this, based on the Mozscape API data? Has anyone ever tried to reverse engineer this metric?
Moz Pro | | brchap0 -
Opensiteexplorer details in API
Is there a way to access opensiteexplorer result details via seomoz's API? Most interested in the social media results
Moz Pro | | fuzzlepop0 -
API callback parameter
I'm using the free API, and would like to use jQuery to fetch the JSON data outputted by the API. I need to make a JSON-P request, and for this I need to be able to pass some form of callback parameter in the url. I'd like to know if this feature is available, and if so what parameter I need to specify in my request. Of course I could just CURL to the API, but that's really a fallback for if I can't get the data using JS. Thanks
Moz Pro | | jgt19850 -
Did the SERP Overlay CSV export disappear?
I'm using Google Chrome with SERP Overlay and it use to say Export to CSV... now it says Get Keyword Difficulty Report. Was this purposely removed? This is one feature that I really liked (SEOQuake does this)... basically where I can export the search results with relevant metrics.
Moz Pro | | WrightIMC0 -
Does Moz track SERP in Google Places?
Hi, how are you? I wanted to ask you if the rankings that Moz shows include the results from Google Places that sometimes appear in the SERPs. Thanks in advance1 Ariel
Moz Pro | | arielbortz0 -
I want to hire someone to write some PHP code using the SEOmoz API.
...but I'm not sure how to go about it. What I need is simple: all I want is to be able to paste a list of URLs (different domains), and have the program return the Page Authority for all those URLs. I understand I can use the free SEOmoz API, particularly the URL Metrics API. Then I want to export the data to an Excel file. That's it. Problem is, I have absolutely no clue how to do it. Obviously I'd pay someone to do it. Pay very well if you can do it professionally and quickly. How can I go about finding the right person for the job? Apologies if this is not the right place to ask this, but I don't know where else to go. Thanks.
Moz Pro | | thegreatpursuit0 -
Any tools for scraping blogroll URLs from sites?
This question is entirely in the whitehat realm... Let's say you've encountered a great blog - with a strong blogroll of 40 sites. The 40-site blogroll is interesting to you for any number of reasons, from link building targets to simply subscribing in your feedreader. Right now, it's tedious to extract the URLs from the site. There are some "save all links" tools, but they are also messy. Are there any good tools that will a) allow you to grab the blogroll (only) of any site into a list of URLs (yeah, ok, it might not be perfect since some sites call it "sites I like" etc.) b) same, but export as OPML so you can subscribe. Thanks! Scott
Moz Pro | | scottclark0