Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Setting up external link goals in GA
I have recently switched all my travel affiliate links to TripAdvisor using their javascript code that converts the natural links (direct links to TA URLs) to affiliate links. I have set up goal tracking in GA to try to put a monetary value on clicks through to these links (based on the known conversion rate to commissionable bookings) but GA is over-reporting the goals by a very large margin. I'm wondering what I've done wrong. The first screen shot below is of the goal settings I used in GA. The second screenshot shows that GA is reporting 7125 goal completions since I set up the goal on 7 August. The third screenshot shows the actual number of GA events recorded on outbound clicks to TA URLs in the same period, which is just 97. So the goal completions being recorded far exceed the actual goal completions, but I'm not sure why. I know I've done something wrong but as this is the first time I've used goals, I'd appreciate some help to see what I've done wrong. Thank you. 3f7dc3de6f9f1373d47994144a5cb016 56aa39130a7d54b07d8f2ce9cf1d7fb0 35ee827275928ca8c4d4c941f0672ef8
Affiliate Marketing | | Gavin.Atkinson0 -
Where can I find best resources for Affiliate Marketing?
Any suggestion for Affiliate training, forums, seminars, etc. for newbie? I'm doing my research but would like to hear from pros. I'm experienced web developer and Adsense publisher, now want to step in the world of affiliate marketing.
Affiliate Marketing | | webplex0 -
Will giving a blogger an affiliate url to my site reduce her links' seo value to me?
If a blogger were to write a post about one of my products and link to my product page, would giving her an affiliate url to link to rather than the normal link spoil the seo value of the link? So instead of mysite.com/product-one it would be mysite.com/affiliate/blogger/product-one Google would think... this is a transaction not a vote of confidence?
Affiliate Marketing | | Brocberry0 -
Can anyone help me find the broadband ISP affiliate program I need?
I have a couple sites that deal with broadband internet access, and I'm looking to monetize beyond adsense. I'd like to sign up for some affiliate programs from internet service providers like Charter, CenturyLink, Comcast, etc. I found a couple affiliate programs but I am having no luck at all figuring out the programs that so many other websites are involved with. In about every case, the user is able to enter in their address and then a search is performed, returning programs available at their location. The programs I've found just do the traditional ad click-throughs for a possible commission. Can anyone enlighten me what kind of programs they are using?
Affiliate Marketing | | bizzer0 -
Best way to start affiliate marketing a new mens magazine website?
I have just started up a new mens magazine website featuring articles, videos, sports news and photos to help drive traffic to the website. I write all the articles myself and do the social media updates in my spare time. I was wondering if anyone has any tips on the best way to start doing affiliate marketing for a new website? I was inspired to do this by successful internet marketers who make a passive income off their websites. So far I have added Amazon and William Hill as affiliate links as I think my target market will be looking for these types of product links. Also, for anyone who has experience in this, how long it taken for your blog to become successful and make money? Many thanks, Alex
Affiliate Marketing | | AlexBarnes0 -
How do the engines look at pages populated using content from an affiliate site?
We are building out a section of our site using an affiliate partnership in the travel space. How would you suggest organizing the content so we do not get penalized for duplicate content? Do you suggest not suing the content at all?
Affiliate Marketing | | TSDigital0 -
Where to look for relevant links
I and many others are targeting keywords such as "free iphone" or "free ps3" as part of incentive marketing schemes that offer rewards for referring new users to businesses. These often seem too good to be true but actually do work (if you don't get caught up in the spam sites). The thing is, what sort of link building should I be looking at once I have my site showing the users how the opportunity works and that it is a legitimate site. I don't want to have free iphone links placed where people may think I'm scamming them, so should I look at press releases, creating infographics or what? I look forward to hearing some of your ideas...
Affiliate Marketing | | GrassRootsSEO0 -
URL Tags / Affiliate tracking help
1. If I give a sales rep a unique tag to put at the end of a URL for tracking purposes, will Google index the "www.mysite.com/service.html?unique-tag" version of a page and display it in search results? 2. Would a tool like Xenu be able to find that version of a URL? Reason for questions: If I give my sales people tags with their names in them, I don't want competitors poaching my sales people by search for all URLs on my domain. Follow up: are there any other potential issues that I should think about? Thanks!
Affiliate Marketing | | pbhatt0