Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Comparison of affiliate marketing programs
Looking for a comparison of major affiliate marketing programs (Amazon Associates vs Commission Junction). Can someone point me in the right direction?
Affiliate Marketing | | nicole.healthline0 -
Affiliate Tool For Competitors Analysis
Hello Guys, I'm Looking for an Affiliate Tool where i can see my competitors affiliate publisher. is there any kind of tool which can help me out to find the best publishers of my competitors? Please Suggest. Thanks Damian Martin
Affiliate Marketing | | CommercePundit0 -
Affiliate Marketing in Australia
Hello Guys My website is BannerBuzz.com.au I'm Looking for Best Affiliate Marketing Websites In Australia and I've Researched and found the following Websites for the same. http://www.clixgalore.com.au/
Affiliate Marketing | | CommercePundit
http://www.jackmedia.com/
http://www.commissionfactory.com.au/ So I Need your valuable opinions on which website I can get maximum exposure in Affiliate Marketing. Please Suggest. Thanks in Advance. Thanks Prabhakar0 -
What impact do affiliate links have on SEO
We commonly reference engineering standards on our site (because half our products are known and searched for by those standard names) and have just signed up as an affiliate to an organisation that sell the standards. Which seems a nice match We're going through and making sure all are no follow now because that seems the general advice - but should we be doing anything else? Yoast says you should mask affiliate links - https://yoast.com/affiliate-links-and-seo/ - but that was from 2010. Matt Cutts is fairly clear on nonfollow - http://www.webpronews.com/heres-what-googles-matt-cutts-says-about-affiliate-links-and-nofollow-2012-06 - but not on whether it affects SERPS Does it? And what is the best way to handle it? This is a typical example of a page where we would have an affiliate link or two. http://www.oakleysteel.co.uk/fake-mill-certificates-en10168
Affiliate Marketing | | Zippy-Bungle0 -
Links exchange - Penguin update potential ??
Hi Everyone, If site A and site B exchange links in the past, it used to be kind of voided. So it is a practicve that we had pretty uch put on the side. But with the new penguin update, do you guys think that if 2 web sites are contacting each other to share a link in the interest of the user that it might be a new best practice to implement? Example, Car site contact car suspension site to exchange a link on relevant content such as blog articles or specific content on a page. What are your thoughts about that ? I am curious to know if anyone is wondering abouot that... Thanks, Alex
Affiliate Marketing | | webit400 -
Affiliate URLs Indexed in Google
We have an active in-house affiliate program which has a create-a-link function where affiliate, as in most affiliate programs, can build links back to our content to produce sales. The problem that I am seeing is that some of the affiliate URL versions are being indexed in Google rather than the original page. For instance.. http://www.ourdomain.com/page_start.aspx?affnum=F025212&Start_Page=RaceCar&referrer=createlink is outranking... http://www.ourdomain.com/racecar This presents 2 issues for us. First, this presents duplicate content issues obviously. Second, we pay our affiliates a portion of the sales so, in order to maximize profitability, we'd like the indexed pages to be the original version. What's the best way to handle this? I want to make sure that affiliates continue to get credit for links from their sites but search engine links should go to the original versions.
Affiliate Marketing | | ATIseo0 -
Linking from High Authority Sites
Dear Mozzes, We're running a property portal for real estate classifieds which receives around 300.000 visits monthly. PA 47 DA 37 Mozrank 5,3 MozTrust 5,8 PR 4 Apart from our real estate classifieds we are producing proprietary trends and statistic, a rare commodity in the country we operate. Luckily we are about to enter into a partnership with one of the leading newspaper portals in the Country, appreciating PAs in the 80-90s. In this coming partnership we will provide: 1. Market statistics, which we can imagine as a chart followed by our logo and a "Powered By", which shall link to our site. 2. An in-framed part of our site in their portal where we list properties for sale, that will carry affiliate strings. In order to for us to get the maximum link value of this high authority site, how should we build the integration and linking? We've looked at similar examples here http://realestate.money.cnn.com/ here http://realestate.money.cnn.com/NY/New_York/ or maybe like the Simply Hired logo here: http://money.cnn.com/ Advices here would be super appreciated. Thanks Eric
Affiliate Marketing | | PropertyPortal0 -
Looking for a nice way to add an amazon affiliate store to my site
We are often asked what our favorite toys, books, resources, etc, are and I want to make a page(s) with our top resources that is a nice clean amazon affiliate store. Are there are good resources anyone knows of to do this? (I'm using a wordpress blog by the way....perhaps there is a plug in?) Thanks.
Affiliate Marketing | | NoahsDad0