Any tools for scraping blogroll URLs from sites?
-
This question is entirely in the whitehat realm...
Let's say you've encountered a great blog - with a strong blogroll of 40 sites.
The 40-site blogroll is interesting to you for any number of reasons, from link building targets to simply subscribing in your feedreader. Right now, it's tedious to extract the URLs from the site. There are some "save all links" tools, but they are also messy.
Are there any good tools that will
a) allow you to grab the blogroll (only) of any site into a list of URLs (yeah, ok, it might not be perfect since some sites call it "sites I like" etc.)
b) same, but export as OPML so you can subscribe.
Thanks!
Scott
-
Not at all. I guess my feeling here is that there is a sort of untapped social graph defined by blogrolls. If it were simple to harvest them upon visiting a blog (e.g. this blogger recommends...) one could do a stumble-on-steroids approach to a niche.
-
I thought you might be able to use the outbound link scraper to grab the outbound link onto the page. Pop in your URLS of the pages you want to scrape and it will spit out our a list of those domaind and urls. You can take those urls and put them into the contact finder and it will return the contact details for those sites. Combine the two spreadsheets for an epiuc list of blogs to contact for your outreach.
This is obviously for link building rather than subscribing - sorry if I have misunderstood what you were trying to do
-
Hi Keri,
That is a very cool tool, but is overkill for this. It takes far too many steps to accomplish only part of the desired goal of grabbing all blogroll URLs (within the blogroll DIV tag) and exporting the list to a valid OMPL file or URL list.
thanks!
-
nothing I saw there would do this. It looks like it could manage to list all external links, and I suppose you could manually pick the blogroll out of it.
-
Hi there,
Well, Keris response reminded me of this question and the fact that I found a tool for scraping these kind of lists:
Here it is (with some other cool tools) , have fun:
-
Hi Scott,
I'm going through older questions. Did you ever find a tool to do what you wanted to do here?
-
One thing to look at is Outwit Hub for Firefox. It might be able to help with that. It can scrape data from a page and do a lot with it. http://www.outwit.com/products/hub/. Don't know that it meets all of your needs, but I also haven't seen a response with anything better at the moment.
-
Hey Scott,
What a great question and <sigh>I don't have the answer. I am going to back to find out what people come up with here. Surely there is someone that lurks these parts that can throw something together?</sigh>
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Crawl 4xx Errors?
Hello! When I check our website's critical crawler issues with Moz Site Crawler, I'm seeing over 1000 pages with a 4xx error. All of the pages that are showing to have a 4xx error appear to be the brand and product pages we have on our website, but with /URL at the end of each permalink. For example, we have a page on our site for a brand called Davinci. The URL is https://kannakart.com/davinci/. In the site crawler, I'm seeing the 4xx for this URL: https://kannakart.com/davinci/URL. Could this be a plugin on our site that is generating these URLs? If they're going to be an issue, I'd like to remove them. However, I'm not sure exactly where to begin. Thanks in advance for the help, -Andrew
Moz Pro | | mostcg0 -
Tools for Monitoring Hundreds to Thousands of Keywords and Rankings
Hi All, I am in process of doing and SEO overhaul for our five global sites in: US, UK, Canada, Sweden, France I'd like to track hundreds of keywords and rankings per site - I'm talking at least 300-400 keywords each site. Each site has its own country domain with both www and www2 domains. So, I need a keyword tool that will let me track massive amounts of keywords. I know that the Moz Pro tool helps, but we only have 350 keywords on this account. I think on this. Any suggestions on something reliable that will provide good data? I'm sure I can get some budget to purchase something, but I also can't spend too too much money. I'm not looking for a massive analytics package. Right now, I'm concerned mainly with our keyword rankings Thanks in advance!
Moz Pro | | CSawatzky0 -
Crawl test from tools
Hi, I notice that the crawl test which is from the Research Tools doesn't really get a new crawl even though there is 2 crawl per day. It will only provide the data which was acquire from the crawl diagnostics in my pro account. There is no point for me to get the data which I get from my crawl diagnostic isn't it? Even seomoz provided with more than 2 crawl per day also useless in this case. This whole thing doesn't make sense as the crawl diagnostics will only perform a full crawl test once every week. but even the crawl test also not helping any thing out for me.
Moz Pro | | hanzoz0 -
What is the Best Local Ranking Tool?
I'm trying to track down a tool that will provide localized rankings within Google Maps/Places, Yahoo Local, Bing Local as well as major local directories such as Yelp, Yellow Pages, etc. Additionally, I'm looking for the results to provide the address being displayed in the ranking. Any suggestions?
Moz Pro | | JonClark150 -
Open site explorer
"Unable to retrieve linking pages on this anchor at this time." This is the notice I get when trying to see links for anchor text. Can someone help?
Moz Pro | | Joseph-Green-SEO0 -
Another link profile tool available
SEOMoz Link Analysis tool apparently doesn't have any info on my clients site - www.tricitymech.com according to the response from Open Site Explorer. Is there a free tool available that doesn't use OSE that you could recommend?
Moz Pro | | DenverKelly0 -
Keyword tool: SEOMOZ spacific month ? vs adword tool 12 month average but same data ???
Running a keyword analysis in SEOMOZ it shows my the folowing information "Local Search Volume (Dec)". I compared the data for the specific country , language and keyword with the adwords keyword tool and it exactly showed me the same numbers. The adwords keyword tool shows: "Local Monthly Searches: This column shows the approximate 12-month average number of search terms matching each keyword" http://support.google.com/adwords/bin/answer.py?hl=en&answer=25148 So if the numbers are the same in google keword tool and SEOMOZ why is SEOMOZ saying that for a specif month? If the data is the same one of both can not be right or probaly I didn't get the point. See screenshot: http://screencast.com/t/GyaaW7EkwV Thanks for help
Moz Pro | | n-media0 -
Page Rank and offline sites
I have a domain with PR6 according to the Historical Pagerank Checker. But that last PR was calculated 2 years ago. I brought the site back online a few days ago and have checked that many/most of the backlinks are still valid. It is now in the Google index but the Historical Pagerank Checker shows PR0. Will it get back its previous rank or something close to it? How long will it take?
Moz Pro | | DomainOptions0