Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blocking external links in Robots.txt - need advice on Best Practice
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt. In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to: a/ leave it as is? b/ allow crawling? Thanks in advance.
Affiliate Marketing | | Ben_Malkin_Develo0 -
What impact do affiliate links have on SEO
We commonly reference engineering standards on our site (because half our products are known and searched for by those standard names) and have just signed up as an affiliate to an organisation that sell the standards. Which seems a nice match We're going through and making sure all are no follow now because that seems the general advice - but should we be doing anything else? Yoast says you should mask affiliate links - https://yoast.com/affiliate-links-and-seo/ - but that was from 2010. Matt Cutts is fairly clear on nonfollow - http://www.webpronews.com/heres-what-googles-matt-cutts-says-about-affiliate-links-and-nofollow-2012-06 - but not on whether it affects SERPS Does it? And what is the best way to handle it? This is a typical example of a page where we would have an affiliate link or two. http://www.oakleysteel.co.uk/fake-mill-certificates-en10168
Affiliate Marketing | | Zippy-Bungle0 -
How to flag inbound affiliate links to Search Engines
Afternoon Mozzers, I was chatting to our VP of marketing earlier about one of our sites with a somewhat unhealthy looking link profile. This is primarily caused by the sector its in and the fact that lots of low quality lead generators / affiliates operate there, sending traffic to us in return for payment on accepted leads. Now, he recalled that there is a way to mark these inbound links, through a URL parameter he thought, as being affiliates and therefore should be ignored as we don’t want them misinterpreted as an attempt to manipulate our rankings. I have been doing a spot of research but can't find a straight answer. heres a few articles i looked at: blogaid, Yoast, Webmaster world and the Moz Q&A.The problem is all these are from the perspective of a site linking out and acting as an affiliate, so all deal with that page not losing PR. The two methods i have derived from this boil down to: No follow the links Use an intermediary redirect page and script which you can block from robots.txt So, back to our case - Are there any ways we can signal to Search engines, as the destination site, that these links are from affiliates (such as this URL parameter our VP had vague recollections off) or should i just get in touch and ask the sites to make them Nofollow? Thanks, Tom.
Affiliate Marketing | | Sarbs0 -
Affiliate site
I have a client I am doing SEO work for. He sells a system that helps people win at video keno. He sells a system for like $200 that shows people step by step how to play and win. Its a good site and he has a ton of verified winnings. My point in saying that, its not a fly-by-night system he wrote in his basement. I know - gambling, not easy for SEO. I don't know why I didn't think of this sooner. Wouldn't this potentially be perfect for affiliate marketing? He offers a good commission to whoever sends him traffic if they buy. If anybody could direct me to a resource or two that is reputable and would handle as much of the set up and management of this I would be very grateful! Thank you! Matthew
Affiliate Marketing | | Mrupp440 -
How to prevent affiliates from bringing negative SEO?
Hi there, I'm just about to integrate iDevAffiliate network for one of my sites, however it occurred to me that affiliates may (unfortunately) resort to black hat methods without me becoming aware of it, which could pass negative link juice to my site. Is there any way to use affiliate networks such as iDev without causing this problem? I was thinking of simply using a different URL, but the problem is that 301 redirects won't mitigate the issue. Thanks!
Affiliate Marketing | | freeunlocks0 -
How do the engines look at pages populated using content from an affiliate site?
We are building out a section of our site using an affiliate partnership in the travel space. How would you suggest organizing the content so we do not get penalized for duplicate content? Do you suggest not suing the content at all?
Affiliate Marketing | | TSDigital0 -
I need some feedback on the value of affiliate aggregators like Link Share
My company decided to start a Link Share affiliate program at the beginning of this year. I've had serious reservations about the value of the program since the beginning but was overruled. Here are my concerns: The majority of publishers seem to be coupon scraper sites or borderline link farms I am told the biggest revenue earners in our network are publishing articles about my company with links back to our site but if they are Google Alerts sure isn't finding them and a manual search hasn't turned up anything either. This makes me think these publishers are using some sort of unsavory way to drive traffic to our site or maybe even cookie traffic that was coming to our site anyway in order to get credit for the sale. Most of all I'd like to hear from anyone else who has used Link Share in the past. Secondly I realize there's not a lot of information here but I'm wondering if this passes the smell test for anyone else. Any feedback would be appreciated.
Affiliate Marketing | | eTundra0 -
How much link juice passes through urls with affiliate id's?
hi we can get a valuable link with the desired anchor text from a news site. the destination url would be something like www.site.com/product. but in order to track conversions, our sales team would like to add an affiliate id to the url, so that it would look like this: www.site.com/product?sess_affiliate=ta how much link juice would a link to this affiliate url pass? would we be shooting our wad by linking to the ?sess-URL instead of the original URL?
Affiliate Marketing | | zeepartner0