Any tools for scraping blogroll URLs from sites?
-
This question is entirely in the whitehat realm...
Let's say you've encountered a great blog - with a strong blogroll of 40 sites.
The 40-site blogroll is interesting to you for any number of reasons, from link building targets to simply subscribing in your feedreader. Right now, it's tedious to extract the URLs from the site. There are some "save all links" tools, but they are also messy.
Are there any good tools that will
a) allow you to grab the blogroll (only) of any site into a list of URLs (yeah, ok, it might not be perfect since some sites call it "sites I like" etc.)
b) same, but export as OPML so you can subscribe.
Thanks!
Scott
-
Not at all. I guess my feeling here is that there is a sort of untapped social graph defined by blogrolls. If it were simple to harvest them upon visiting a blog (e.g. this blogger recommends...) one could do a stumble-on-steroids approach to a niche.
-
I thought you might be able to use the outbound link scraper to grab the outbound link onto the page. Pop in your URLS of the pages you want to scrape and it will spit out our a list of those domaind and urls. You can take those urls and put them into the contact finder and it will return the contact details for those sites. Combine the two spreadsheets for an epiuc list of blogs to contact for your outreach.
This is obviously for link building rather than subscribing - sorry if I have misunderstood what you were trying to do
-
Hi Keri,
That is a very cool tool, but is overkill for this. It takes far too many steps to accomplish only part of the desired goal of grabbing all blogroll URLs (within the blogroll DIV tag) and exporting the list to a valid OMPL file or URL list.
thanks!
-
nothing I saw there would do this. It looks like it could manage to list all external links, and I suppose you could manually pick the blogroll out of it.
-
Hi there,
Well, Keris response reminded me of this question and the fact that I found a tool for scraping these kind of lists:
Here it is (with some other cool tools) , have fun:
-
Hi Scott,
I'm going through older questions. Did you ever find a tool to do what you wanted to do here?
-
One thing to look at is Outwit Hub for Firefox. It might be able to help with that. It can scrape data from a page and do a lot with it. http://www.outwit.com/products/hub/. Don't know that it meets all of your needs, but I also haven't seen a response with anything better at the moment.
-
Hey Scott,
What a great question and <sigh>I don't have the answer. I am going to back to find out what people come up with here. Surely there is someone that lurks these parts that can throw something together?</sigh>
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links being reported in Webmaster Tools
Hi Are the Total Links To Your Site, as reported in GWT, purely external inbound links ? Since these links are usually, as far as i can tell, much higher in number than any other link reporting tool and hence, i presume, more accurate, why don't services such as Moz etc include this in reporting ? I know its just a total number and link quality is whats important not quantity, but i would have thought interesting to show in reporting in conjunction with link quality info such as is already reported. Since most backlink reporting tools do show a total but always much much lower than that reported in gwt (i think) All Best Dan
Moz Pro | | Dan-Lawrence0 -
Magento: Moz finding URL and URL?p=1 as duplicate. Solution?
Good day Mozzers! Moz bot is finding URL's in the Catalogue pages with the format www.example.com/something and www.example.com/something?p=1 as duplicate (since they are the same page) Whats the best solution to implement here? Canonical? Any other? Cheers! MozAddict
Moz Pro | | MozAddict0 -
Can Moz tools help me with this effort?
I want to brainstorm different keywords, look for high search volumes, and low competition. Then I want to create landing page, rank them using seo techniques, and collect optin email addresses so I can communicate with interested users and build helpful products. With all the moz tools available to us, how can I accomplish the above mentioned goal? Have you done some thing similar? What are your experiences? Am I talking pie in the sky? Are there any practical examples where all these steps were executed? Thanks
Moz Pro | | zsyed0 -
Looking to Hire an SEO Expert for Travel Site
Hello Everyone, I hope this is OK to post here. I run a travel website and we operate tours abroad. We are slowly expanding after nearly 10 years in the field. I started the company from scratch and currently do my own website design and SEO work. However, over the last couple of years I've lost the SEO connection due to being so darn busy. I started account here with SEO Moz in order to get the ball rolling again. We continually rank high for keywords, publish superb content by great writers, and overall are doing fine. I would just like to ensure this keeps happening and that we have an expert on board to help us in sustainable organic SEO practices. What I'd like to do is hire an SEO expert to help with our site. It might be part time (10 or so hours per week, perhaps?) Anyhow, I would like if this person used SeoMoz and really had their fingers on the pulse of the SEO world. I want all white-hat SEO, help with page optimization, internal link structure, and help getting quality external links. I know that over optimization is something that is now on the forefront of SEO work, so of course keeping this in mind helps. I would like to be kept up to date (perhaps weekly overviews of data and how we are doing) and am willing to make changes as quickly as needed on the site. I would like to see how we rank with competitors and what we need to do to stay on top - for the long term (no short sighted decisions). I want to learn as we progress, not because I want to do the SEO work myself, I just enjoy knowing what is going on. I will not micromanage. I am too busy for that : ) I am not sure what amount SEO experts are paid, but I have a slightly flexible budget. I really want someone will communicate when needed and is thorough in their responses. Attention to detail is a must. I would prefer an SEO expert located in the US but who also knows how to access / think about markets abroad (Canada/UK/Australia). I suppose we can discuss pay once I have a shortlisted group. I am looking to hire and work with someone starting pretty much immediately (even in May). For starters, I suppose our budget might be $500 to $1500 per month. For the first month or so, we might offer payment on a stipend basis just so we can ensure whoever is hired is the perfect fit. It would be great if this person had worked on travel sites before or has at least traveled abroad a few times. But, no worries either way. Thanks so much! Thomas at jbt@journeybeyondtravel.com
Moz Pro | | journeybeyondtravel0 -
On page links tool here at Seomoz
Hi Seomoz - first of all, thanks for the best SEO tools I have ever worked with (this is my first question in this forum, and also I just subscribed as a paying customer after the 30 days trial you guys offer). My question: After having worked for several weeks on getting the numbers of links in our forum on www.texaspoker.dk down, we are somewhat surprised to see that we didn't succeed in getting lower numbers. For instance, this page: http://www.texaspoker.dk/forum/aktuelle-konkurrencer/coaching-projekt-bliver-du-den-udvalgte has (that's what Seomoz seo tool tells us): 239 on page links. Can this really be true? We can't find these links, and we actuually did a lot to lower the numbers of links, for instance the forum members picture was a link before, and also there was a "go to top" link in each post in the forum. Thanks a lot.
Moz Pro | | MPO0 -
Is the keyword difficulty tool the most helpful in all situations?
I understand that the scores it generates are essentially based on the difficulty of appearing on the first SERP for the keyword in question. That said, I am having a lot of difficulty finding keywords in my niche which return a score that would make this easily achievable for a site of my size.... The reason I'm pointing this out is because theoretically, a keyword could have a HIGHLY competitive first SERP, with a significant drop-off on the second SERP, which would make achieving a top ranking on that page substantially easier. So my question really is, is the importance of appearing on the first SERP so unequivocally important that it is a pointless activity to attempt deliberately to rank for keywords on the second SERP, which is ignored by the keyword difficulty tool? I know the breakdown of clicks goes something like 40% for top spot, 12% for second and downwards from there, but if a certain query has over a million searches per month, for example, it would still be possible to get considerable amounts of traffic by trying to rank highly on the second SERP, which the keyword difficulty tool cannot help with. So is this really a useless activity?
Moz Pro | | ZakGottlieb710 -
On page optimisation tool issues
When viewing my campaign and looking at the on page optimisation tool, I have a few issues. I seems to only shows the keywords I want rankings for and how optimised my homepage is for those keywords. Is there any way I can get it to analyse permanently specifc keywords for specific pages because my homepage isnt optimised for some keywords which are on my list, which I have optimised other pages for, and because its looking at my homepage its getting a really low grade, and looks really bad and frustrates me because I cant work this out. Any help greatly appreciated.
Moz Pro | | CompleteOffice1 -
On-Page Optimisation tool on intranet pages
Does anybody know if there's any easy way to use the On-Page Optimisation tool on intranet or not publicly accessible pages? Thanks!
Moz Pro | | neooptic0