Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Setting up external link goals in GA
I have recently switched all my travel affiliate links to TripAdvisor using their javascript code that converts the natural links (direct links to TA URLs) to affiliate links. I have set up goal tracking in GA to try to put a monetary value on clicks through to these links (based on the known conversion rate to commissionable bookings) but GA is over-reporting the goals by a very large margin. I'm wondering what I've done wrong. The first screen shot below is of the goal settings I used in GA. The second screenshot shows that GA is reporting 7125 goal completions since I set up the goal on 7 August. The third screenshot shows the actual number of GA events recorded on outbound clicks to TA URLs in the same period, which is just 97. So the goal completions being recorded far exceed the actual goal completions, but I'm not sure why. I know I've done something wrong but as this is the first time I've used goals, I'd appreciate some help to see what I've done wrong. Thank you. 3f7dc3de6f9f1373d47994144a5cb016 56aa39130a7d54b07d8f2ce9cf1d7fb0 35ee827275928ca8c4d4c941f0672ef8
Affiliate Marketing | | Gavin.Atkinson0 -
Affiliate site
I have a client I am doing SEO work for. He sells a system that helps people win at video keno. He sells a system for like $200 that shows people step by step how to play and win. Its a good site and he has a ton of verified winnings. My point in saying that, its not a fly-by-night system he wrote in his basement. I know - gambling, not easy for SEO. I don't know why I didn't think of this sooner. Wouldn't this potentially be perfect for affiliate marketing? He offers a good commission to whoever sends him traffic if they buy. If anybody could direct me to a resource or two that is reputable and would handle as much of the set up and management of this I would be very grateful! Thank you! Matthew
Affiliate Marketing | | Mrupp440 -
Affiliate marketing dead?
I have created a network of travel websites that I have decided to promote little by little.. Also so they getdomain age and google trust.... but now when you search for all my main keyphrases... you either have wikitravel,lonelyplanet,frommers and other big brand. I guess that Google gave them better value as to get trusted travel information... However, it seems it will be very difficult to rank my sites. I could do long tail keyphrases but again might not be worth it... I used to work as a SEO consultant for other travel sites selling products... they still rank pretty well as they are sites of value selling something.. of course threy compete with other sites but we are in the same level. How can you compete with loneyplanet with your affiliate site because I am not selling anything on my sites. I am pretty much desperate. I designed a network of 30 sites... all good content with good SEO on the page... of course, I will be honest. I might have to change the content to make it better... you can check one of my site here; www.simplyparis.org it is worth it or better stop this project and focus on 1 product oriented website ? Is anyone working as an affiliate marketer with experience after google panda and pinguin ? is affiliate marketing dead in 2013 now?
Affiliate Marketing | | sandyallain0 -
Please recommend an affiliate program
I'm a bit surprised there doesn't seem to be a lot of info on their already on SEOMoz, so I'd appreciate any insight into this for a client of mine who is a currently a startup There are lots of options from massive brand names like Google and Amazon right down to countless services I've never heard before. For a smaller start-up with a technical edge. specifically service subscriptions for the webcam / video camera type of market, what would be an effective and cost friendly options to explore? Any shared experience is very appreciated. Here is a list that pretty well captures the top 10, so these are all under consideration: http://www.problogger.net/archives/category/affiliate-programs/ Cheers, Chris
Affiliate Marketing | | cbubinas0 -
Recommended affiliate programmes (particularly UK-based)?
I'm starting a photography/travel/music blog and thought it might be a good idea to stick the occasional affiliate link in when it's appropriate. Does anyone have any recommendations? The related items would mainly be to DVDs, books and music, and sometimes something more travel specific like flights. Are the programmes from the likes of Amazon and ebay good, or would it be better to sign up to a system that covers many companies? It'll be my own CMS if that makes any difference. Thanks. 🙂
Affiliate Marketing | | Alex-Harford0 -
Will giving a blogger an affiliate url to my site reduce her links' seo value to me?
If a blogger were to write a post about one of my products and link to my product page, would giving her an affiliate url to link to rather than the normal link spoil the seo value of the link? So instead of mysite.com/product-one it would be mysite.com/affiliate/blogger/product-one Google would think... this is a transaction not a vote of confidence?
Affiliate Marketing | | Brocberry0 -
URL Tags / Affiliate tracking help
1. If I give a sales rep a unique tag to put at the end of a URL for tracking purposes, will Google index the "www.mysite.com/service.html?unique-tag" version of a page and display it in search results? 2. Would a tool like Xenu be able to find that version of a URL? Reason for questions: If I give my sales people tags with their names in them, I don't want competitors poaching my sales people by search for all URLs on my domain. Follow up: are there any other potential issues that I should think about? Thanks!
Affiliate Marketing | | pbhatt0 -
How much link juice passes through urls with affiliate id's?
hi we can get a valuable link with the desired anchor text from a news site. the destination url would be something like www.site.com/product. but in order to track conversions, our sales team would like to add an affiliate id to the url, so that it would look like this: www.site.com/product?sess_affiliate=ta how much link juice would a link to this affiliate url pass? would we be shooting our wad by linking to the ?sess-URL instead of the original URL?
Affiliate Marketing | | zeepartner0