Any tools for scraping blogroll URLs from sites?
-
This question is entirely in the whitehat realm...
Let's say you've encountered a great blog - with a strong blogroll of 40 sites.
The 40-site blogroll is interesting to you for any number of reasons, from link building targets to simply subscribing in your feedreader. Right now, it's tedious to extract the URLs from the site. There are some "save all links" tools, but they are also messy.
Are there any good tools that will
a) allow you to grab the blogroll (only) of any site into a list of URLs (yeah, ok, it might not be perfect since some sites call it "sites I like" etc.)
b) same, but export as OPML so you can subscribe.
Thanks!
Scott
-
Not at all. I guess my feeling here is that there is a sort of untapped social graph defined by blogrolls. If it were simple to harvest them upon visiting a blog (e.g. this blogger recommends...) one could do a stumble-on-steroids approach to a niche.
-
I thought you might be able to use the outbound link scraper to grab the outbound link onto the page. Pop in your URLS of the pages you want to scrape and it will spit out our a list of those domaind and urls. You can take those urls and put them into the contact finder and it will return the contact details for those sites. Combine the two spreadsheets for an epiuc list of blogs to contact for your outreach.
This is obviously for link building rather than subscribing - sorry if I have misunderstood what you were trying to do
-
Hi Keri,
That is a very cool tool, but is overkill for this. It takes far too many steps to accomplish only part of the desired goal of grabbing all blogroll URLs (within the blogroll DIV tag) and exporting the list to a valid OMPL file or URL list.
thanks!
-
nothing I saw there would do this. It looks like it could manage to list all external links, and I suppose you could manually pick the blogroll out of it.
-
Hi there,
Well, Keris response reminded me of this question and the fact that I found a tool for scraping these kind of lists:
Here it is (with some other cool tools) , have fun:
-
Hi Scott,
I'm going through older questions. Did you ever find a tool to do what you wanted to do here?
-
One thing to look at is Outwit Hub for Firefox. It might be able to help with that. It can scrape data from a page and do a lot with it. http://www.outwit.com/products/hub/. Don't know that it meets all of your needs, but I also haven't seen a response with anything better at the moment.
-
Hey Scott,
What a great question and <sigh>I don't have the answer. I am going to back to find out what people come up with here. Surely there is someone that lurks these parts that can throw something together?</sigh>
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site visitors dropped off- have tried everything
Hello Moz team....I love the tools but I'm more of a content created for my web site than a developer. So I am super stuck with a major money losing problem. My traffic has nearly gone to zero and I have good content, regular content, no duplicate content, and so on. Can someone help.... Today I disavowed thousands of bad inbound links found in a Moz report Since January I have been fixing pages with missing meta data found in the crawl report In December 2018 I switched over to HTTPS I have no clueeeeeee
Moz Pro | | MattBanes0 -
MoZ & Other Keyword Tools
Hi There, 1- Does MoZ provide the data on “which keyword searchers are searching in a particular market”, so that I can make a better decision regarding my own keywords?If MoZ does not do this what other best tool is out there in market for this particular purpose? 2- Suppose I decide some keywords for my site and start doing optimization for them. "Does MoZ tell me “how much traffic my those keywords are receiving with the passage of time”. If MoZ does not do this what other best tool is out there in market for this particular purpose? 3- Does Moz give "data on competitors keyword activity also"? If MoZ does not do this what other best tool is out there in market for this particular purpose? Hope somebody with concrete knowledge and experience will enlighten me. Cheers Tanveer
Moz Pro | | Sequelmed0 -
I am experimenting with Raven Tools and have a question
Since I am an in-house SEO I'm a little slower to adopt some of the paid tools out there, but I am thinking of jumping in and opting for the paid version of Raven Tools. Can anyone who's currently using Raven Tools suggest how and where is the best place to export backlinks info for import into the Link Manager? I want something that will give me as much of the data as possible already filled in (i.e. PR, domain authority, anchor text, follow/no follow information etc.). I don't mind if I have to rearrange it, I just don't want to have to be inputting things like PR and anchor text one at a time by hand. I do have SEOMoz pro and I see that I can pull a CSV of backlinks and do two separate reports, one for followed and one for not followed links, etc. I'm just wondering if anyone has any additional suggestions. How about Ahrefs.com? Is anyone using the paid version of that tool in conjunction with Raven Tools for Link Management? Is there a better tool out there for Link Management than Raven Tools? (I just found this link in a Q & A thread for an open source SEO panel to try...http://www.seopanel.in/ - anyone using that one?) Thanks all!
Moz Pro | | danatanseo0 -
Could SEOMOZ make a tool thats connected directly to Webmaster Tools?
Could SEOMOZ make a tool that Crawls Internal Links that's connected (or crawls) directly from Webmaster Tools? (So link metrics on our domains would get updates more often) This would eliminate the cost of crawling the entire internet. I hate having to wait a month to see updates on my domain metrics and links. I would like to see updates regularly on my own domain. I understand you can just look at Webmaster tools to see your latest links but they don't have any link metrics. So it’s difficult to see your best links. If this already exists please refer me? Otherwise please show your support for this being developed?
Moz Pro | | charles10 -
What tools should i use to research the following site
Hi i am researching the following site http://www.telegraph.co.uk/travel/ and i would like to know what keywords they are using to gain traffic and i would like to know what tools i need to use to research how they are achieving great success with google. i am new to the semoz tools so i would like help in using them to study this site and other sites, can anyone please let me know what i need to do and what tools i need to use to research this site and find the links to the site and how they are achieving great success many thanks
Moz Pro | | ClaireH-1848861 -
Site metric tool
Is there a tool on here (or anywhere else) where you can upload a list of sites (say 500 sites) and be given things like mozrank on it?
Moz Pro | | thefresh0 -
Open Site Explorer Data
Good afternoon SEOmoz, I first want to congratulate with you for the amazing new look and new features of Open Site Explorer, it's an amazing and valuable tool that I use every day and that has helped me build tons of links for my clients. I have been keeping track of the OPE data on a monthly base and I just noticed something strange. In May 2011, OSE was showing that my client's website had 12,624 links, in July 2011 the website had 14,157 links and today I just ran a new report that showed just 6,401 links. I don't want to believe that in 2 months we lost half of the links, so I was wondering if you guys can give me more insights about this data.
Moz Pro | | Aviatech
Did SEOMoz change the way they measure the number of links? Or is this just a bug or issue they are having with the new version of OSE? Thanks!0 -
A tool for seeing all the keywords a website ranks for
As title, is there a tool out there that will allow you to see all the keywords a website ranks for in the first few pages of a particlur search engine? I would be particularly interested for keywords in the first 10 pages of Google. Thanks
Moz Pro | | blagger0