Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do you find Affiliate Links on your site that have not been nofollowed?
We've just signed up to an affiliate scheme because we were sending links to them because we thought their product was valuable to our users. So we now have to go through and nofollow all of these links over 100's of pages. Is there any way that do a crawl of the site to identify all links to a particular site and tell me what page they are on and whether they are nofollow/follow?
Affiliate Marketing | | Zippy-Bungle0 -
Advise about Social Sharing between two loosely affiliated website - (retweets/fb cross sharing)
So basically, I have two websites. One being my Primary website and another which is very Niche/Seasonal. Now the Niche website received a penalty, and was later re-included however google traffic has been extremely limited since being reindexed. (bing never had a problem, still top ranked) I don't directly link back and forth between the two websites because its not worth hurting my primary websites ranking. I'm sure google knows I own both anyway, as they are on the same webmaster tools but the Niche site was kicked for spammy link building & its been an endless process removing them all. Anyway, with that being my concern. Would it be acceptable to cross share facebook posts, retweet and do things like that on occasion? My primary reason for asking is because the Niche website has a very large social following, so I would like to share post/tweets from my Primary site when its relevant and vice versa. Appreciate any suggestions/comments. Thanks!
Affiliate Marketing | | Southbay_Carnivorous_Plants0 -
Merchant´s data feed for affiliates is the same content as their own website...
Hi Some advice appreciated. Started working on a site and found out that they are giving their unique content to their affiliates (an XML feed so appearing on another domain). In this case, if they want to provide the data like that, how can we protect ourselves? Should we use author tags in our html, is that necessary? Is there any fix other than "stop doing that and give them different content"? Thanks
Affiliate Marketing | | xoffie0 -
Affiliate program in the education niche
Just curious if anyone knew of some good ones out there. Looking to try a couple out. Thanks!
Affiliate Marketing | | astahl110 -
How many affiliate links is considered too many?
Hi, Let's say you have great reviews for 50 products and some of these products do have affiliate links on review pages. And then you have user scores and you come up with a top 20 product list sorted by user scores. Now if you have the list of top 20 products on one page and all these products have an affiliate link (with nofollow relation and a 301 redirect) on the same page with only a couple of images and a summary of the review linked to the review pages, would this still be considered as what Google calls a "bridge page"? Would it be better to still generate the top 20 list but rather link to review pages only? (to avoid too many affiliate links on one single page).
Affiliate Marketing | | Gamer070 -
Limit of links in the same page
Hi, I have some pages that have more than 100 links. They are links to some stores (affiliate links). I set all of them as rel="nofollow" but I think Google is penalizing me because the link quantity. What is a good number of URL in the same page? Thanks
Affiliate Marketing | | GDB0 -
Does an affiliate link bring the same SEO juice as a standard link?
I wonder if affiliate links, like the ones offered by Amazon Associates program bring the same SEO as would a link to the same page without this additional "ref=..."?
Affiliate Marketing | | maciek-0 -
Affiliate network options?
I'm looking, for an affiliate network solution, that can provide not only conversions through affiliate marketing alone, but also as linkbait. What kind of tools do we have available? i remember reading a SEOMoz Blog about the best SEO tools of 2010, and i saw some affiliate options there. Looking to hear SEO community opinions on options like this.
Affiliate Marketing | | Blargh1230