Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Affiliate program links
Hello, I've question regarding affiliate partner generated links.Company www.example.com has a affiliate program. Partner www.example2.com links to the company site with affiliate link www.example.com/?affid=xxxx in banner, article etc. As I understand these type of links could hurt our Company site, so I asked partners to use rel="nofollow" in all links where they link to us usign ?affid=xxxx in url. To be 100% sure that I did the right thing I decided to post here my question. Could these type of links hurt our Company site? Regards,
Affiliate Marketing | | juris_l
Juris0 -
Links exchange - Penguin update potential ??
Hi Everyone, If site A and site B exchange links in the past, it used to be kind of voided. So it is a practicve that we had pretty uch put on the side. But with the new penguin update, do you guys think that if 2 web sites are contacting each other to share a link in the interest of the user that it might be a new best practice to implement? Example, Car site contact car suspension site to exchange a link on relevant content such as blog articles or specific content on a page. What are your thoughts about that ? I am curious to know if anyone is wondering abouot that... Thanks, Alex
Affiliate Marketing | | webit400 -
Duplicate tracking of Affiliate and Organic orders
I started diving into the over lap between Organic and affiliate orders and can see that on average about 10% of my organic orders get tracked as both affiliate and organic (meaning i am paying affiliate commissions on them). I want to be able to better account for the different types of order and where they are coming from. My question is simply if anyone has any ideas as to how better to figure out if those orders should be canceled on the affiliate end or if there is a better way in analytics to see where the order is actually getting attributed to through some type of goal or advanced set up or something (the download of orders from organic that I compared to the affiliate ones was taken from Ecommerce > Transactions > Medium (Organic) then I compared that to the order numbers from the affiliate company I am using). Hope this makes sense and thanks for any information that can help with this!
Affiliate Marketing | | Gordian0 -
Will giving a blogger an affiliate url to my site reduce her links' seo value to me?
If a blogger were to write a post about one of my products and link to my product page, would giving her an affiliate url to link to rather than the normal link spoil the seo value of the link? So instead of mysite.com/product-one it would be mysite.com/affiliate/blogger/product-one Google would think... this is a transaction not a vote of confidence?
Affiliate Marketing | | Brocberry0 -
Which affiliate programs pass the best SEO value through their links?
How good is Google at detecting the affiliate nature of those links? How does this rate as a linkbuilding strategy?
Affiliate Marketing | | menachemp0 -
Affilate Links
Hi I’m looking for confirmation of my worries about affiliate links to a website I'm working for and hopefully get some ideas of how to solve the problem of having affiliates without losing the strength of links from affiliate web sites. Poplidays.com has an affiliate program with an independant affiliation platform, but it also has a couple of partners who link directly to Poplidays.com from their sites. Poplidays wants to reward the partner sites in the same way as it does affiliates: with a percentage of sales generated by visitors. However as their SEO I don’t want to lose these links. They are from established, quality sites in the same subject area and, in my opinion, have a lot of SEO value. Affiliate links are 302 redirects from urls on the affiliation platform’s domain; something like http://poplidays.affiliation.com/?clientid=123 to www.poplidays.com Can you confirm that replacing a direct link to www.poplidays.com with a 302 redirect of this type will be a bad for Poplidays rankings? Any proof of this from a reliable source that I can quote? The affiliation platform is insisting that there is no negative effect and that no one has ever made a remark like mine before. Any good ideas of how to resolve the problem of rewarding linkers without spoiling the links for SEO? Bearing in mind that Poplidays want to be able to reward sites for visitors who may buy on a later interaction with the site (up to 30-days) and want an automatic report so that the partner can follow his commissions. I’ve tried implementing something through Google Analytics but it wasn’t perfect. Your in house affiliation system is working with links like http://go.seomoz.org/aff_c?offer_id=1&aff_id=1624&url_id=1&file_id=24 that 302 to http://www.seomoz.org/features?affid=1624&txnid=10274208218121120514 Is this a better / perfect solution ? Compared to a direct link to http://www.seomoz.org/features ? Thanks - Neil
Affiliate Marketing | | CarmenImmobilier0 -
Hiding Affiliate Links
Hello, Currently I'm developing a website that will contain a lot of affiliate links (5000). This is because I'm going to compare the prices of a lot of products. The visitor can go to all websites (my advertisers) that sell these products via affiliate links. My question is how to hide these affiliate links. I've read lots of solutions, like: cloaking, rel=nofollow, using JavaScript, 301 redirects. I would like to know what the best solution is to deal with these affiliate links for SEO. Regards, Martijn
Affiliate Marketing | | MartijnWijbenga290 -
Does an affiliate link bring the same SEO juice as a standard link?
I wonder if affiliate links, like the ones offered by Amazon Associates program bring the same SEO as would a link to the same page without this additional "ref=..."?
Affiliate Marketing | | maciek-0