Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My affiliate links are being crawled by Google. Is this a bad thing? Also, should I add rel: nofollow to affiliate links?
Hi All, I have an issue on Google Search Console. It seems all my affiliate links are being crawled. Is this a bad thing? The reason I found out is that all but one link was flagged with a warming. It goes on to say that it was blocked by robots.txt. The strange thing is that I've not set any links to be blocked in robots.txt. Also, should affiliate links be nofollow? I'm asking because I was recently sent this article by Yoast on how to properly cloak my affiliate links without using a plugin. https://yoast.com/cloak-affiliate-links/ In the method, they apply the nofollow and noindex rules to the links. I'm just a little confused on how to go about affiliate links the SEO friendly way.
Affiliate Marketing | | nomad_blogger0 -
Person using expired domain and its links to drive traffic
Hi, I know about people using expired domains to drive juice to their primary site but what about people using AN expired domain as their primary site (totally changing that site into a trashy affiliate-marketing vehicle)? The site I'm looking at is thegunzone.com. It has, according to Semrush, almost 38K links. It used to be a legit 17-year-old firearms hobby site, and this is what it originally looked like: http://web.archive.org/web/20120213184627/http://thegunzone.com:80/ Here is its last page before it closed and the domain purchased by the affiliate marketer: http://web.archive.org/web/20170315084035/http://www.thegunzone.com/ It closed around February of 2017, and some affiliate marketer bought it and all its backlinks. However, all those backlinks, which were previously to various articles, are now directed back to those articles (which don't exist anymore) but the homepage, including Wikipedia links. Here's an example: https://en.wikipedia.org/wiki/Polygonal_rifling At the bottom, in the 7th Reference, there's a link to an article called " "Learning About Shooting . . ." but if you click on the original link, it just goes to thegunzone.com homepage. Again, the site's totally different. And there are just thousands of such backlinks to former articles that don't exist anymore but are redirected to this schlocky site's homepage (and it's passing its juice through too). My question is this: this cannot be kosher with Google backlinking policies, right? Is this prevalent on the internet? Why hasn't thegunzone.com been found out and its rankings penalized yet? And how do I report him? I see tons of other sites using this basic strategy too on search results with various hunting keywords. (Disclosure: I do own a hunting/firearms blog, but I don't do any backlinking at all.) Any help would be sincerely appreciated.
Affiliate Marketing | | HandyWoman1 -
Is it a good idea to make HubPages? Or Squidoo lens?
Hello, I would like to promote the products in http://www.giftbasket4kids.com with Hubpage and or Squidoo.com. we are starting with Christmas Items. Please share your ideas as Should we use Hubs or Squidoo Presently he website is under reconsideration with Google for Penguin hit. We have lost most of the keywords on three landing pages. We do not have a lot of time before Christmas sales. How fast can we publish the pages? What are to do and what not to do list. I appreciate all your comments. Thanks in advance.
Affiliate Marketing | | giftbasket4kids0 -
Limit of links in the same page
Hi, I have some pages that have more than 100 links. They are links to some stores (affiliate links). I set all of them as rel="nofollow" but I think Google is penalizing me because the link quantity. What is a good number of URL in the same page? Thanks
Affiliate Marketing | | GDB0 -
Link Building Intricacies
I'm about to embark on link building campaign. I have several key phrases that I want to develop such as "health & fitness products" and "vitamins & supplements". My site is specifically targeted at Australia men. Should I also aim to include the words "Australia" and "men" in my link anchor text? Or should leave the qualifiers to the title tag for my destination url? I'm also wondering, by having "health & fitness products" as my link anchor text and the same text in my title tag for my destination url, will that build value for the various keyword variations in my keyword phrase such as "health & fitness" and "fitness products"? Thanks!
Affiliate Marketing | | B-man0 -
How should I deal with the affiliate URLs we create through our affiliate program?
Our website offers an affiliate program and every affiliate gets a link http://www.ourwebsite.com/?affiliateID. Because we get many such links we use redirects to avoid duplicate content. Unfortunately we have 302 redirects set up and I want to make them 301. Many of our affiliates build backlinks using their http://www.ourwebsite.com/?affiliateID as the URL. If such links were redirected through a 301, would they pass any juice? Can our site get penalized by search engines for having backlinks to URLs that contain affiliate IDs, URLs that are 301 redirected to our homepage lets say?
Affiliate Marketing | | nick.leech0 -
Does an affiliate link bring the same SEO juice as a standard link?
I wonder if affiliate links, like the ones offered by Amazon Associates program bring the same SEO as would a link to the same page without this additional "ref=..."?
Affiliate Marketing | | maciek-0 -
Gaining juice from affiliate links
Afternoon Mozzers, We run a basic affiliate program whereby we distribute banners and tools to affiliate sites who then link back to our homepage via these, often through a tool that incorporates an HTML link with our primary keyword term embedded. This banner/tool/link heads to our homepage, and will then incorporate that particular affiliate's unique code in the URL, eg. www.example.com/1A-265-GNH Seems pretty standard practice so far? So, considering this, that we include our primary keyword term in HTML in some banners, some also link to our homepage via an embedded image (and others then just work off cookies as per usual), can anyone hazard a guess, or know for certain, the beneficence of this as a link building means? I ask in terms of trying to determine whether to continue punting our tools and banners as a decent link building opportunity, or concentrate more on other link building means. Hope I've been clear enough, but am happy to elaborate if need be. Cheers
Affiliate Marketing | | Martin_S0