Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Person using expired domain and its links to drive traffic
Hi, I know about people using expired domains to drive juice to their primary site but what about people using AN expired domain as their primary site (totally changing that site into a trashy affiliate-marketing vehicle)? The site I'm looking at is thegunzone.com. It has, according to Semrush, almost 38K links. It used to be a legit 17-year-old firearms hobby site, and this is what it originally looked like: http://web.archive.org/web/20120213184627/http://thegunzone.com:80/ Here is its last page before it closed and the domain purchased by the affiliate marketer: http://web.archive.org/web/20170315084035/http://www.thegunzone.com/ It closed around February of 2017, and some affiliate marketer bought it and all its backlinks. However, all those backlinks, which were previously to various articles, are now directed back to those articles (which don't exist anymore) but the homepage, including Wikipedia links. Here's an example: https://en.wikipedia.org/wiki/Polygonal_rifling At the bottom, in the 7th Reference, there's a link to an article called " "Learning About Shooting . . ." but if you click on the original link, it just goes to thegunzone.com homepage. Again, the site's totally different. And there are just thousands of such backlinks to former articles that don't exist anymore but are redirected to this schlocky site's homepage (and it's passing its juice through too). My question is this: this cannot be kosher with Google backlinking policies, right? Is this prevalent on the internet? Why hasn't thegunzone.com been found out and its rankings penalized yet? And how do I report him? I see tons of other sites using this basic strategy too on search results with various hunting keywords. (Disclosure: I do own a hunting/firearms blog, but I don't do any backlinking at all.) Any help would be sincerely appreciated.
Affiliate Marketing | | HandyWoman1 -
What's the best way to go about making Duplicate Content Pages on my domain for affiliates?
Hello, I would like to make a bunch of duplicate pages of my site's Home Page, that way affiliates of mine can have a page of their own with links specialized to themselves littered throughout the page. What's the best practice going about this without jeopardizing domain authority from tons of duplicate content signals firing off?
Affiliate Marketing | | Benavest0 -
Affiliate Program: Does anyone know how to set this up?
Hi Guys Does anyone know how I can make a affiliate programme on my website eg - I have a sales person in two of our local regions and I want to find a way I can track any sales made through them via my website so I can pay them their commission. Any advice around this, the website is wordpress! Cheers Justin
Affiliate Marketing | | edward-may0 -
Affiliate program links
Hello, I've question regarding affiliate partner generated links.Company www.example.com has a affiliate program. Partner www.example2.com links to the company site with affiliate link www.example.com/?affid=xxxx in banner, article etc. As I understand these type of links could hurt our Company site, so I asked partners to use rel="nofollow" in all links where they link to us usign ?affid=xxxx in url. To be 100% sure that I did the right thing I decided to post here my question. Could these type of links hurt our Company site? Regards,
Affiliate Marketing | | juris_l
Juris0 -
Affiliate URLs Indexed in Google
We have an active in-house affiliate program which has a create-a-link function where affiliate, as in most affiliate programs, can build links back to our content to produce sales. The problem that I am seeing is that some of the affiliate URL versions are being indexed in Google rather than the original page. For instance.. http://www.ourdomain.com/page_start.aspx?affnum=F025212&Start_Page=RaceCar&referrer=createlink is outranking... http://www.ourdomain.com/racecar This presents 2 issues for us. First, this presents duplicate content issues obviously. Second, we pay our affiliates a portion of the sales so, in order to maximize profitability, we'd like the indexed pages to be the original version. What's the best way to handle this? I want to make sure that affiliates continue to get credit for links from their sites but search engine links should go to the original versions.
Affiliate Marketing | | ATIseo0 -
In what ways would an affiliate site be okay with Google?
I know Google has slammed affiliate sites hard. But affiliate marketing can be a real business and had an idea for one, but I would like to know if ALL affiliate sites are looked at as evil in Google's eyes, or if it's just those with thin and duplicate content in the product descriptions. I know they want brands to come up ahead of affiliates of that brand, and that makes sense. If I sold Acme Widgets products, Acme Widgets itself should come up ahead of my site. But what if the site sold widgets in general, with the term widgets not a branded keyword? If all product descriptions are unique and well-written content, on a site that is high quality, would it still be downgraded by Google just because there are affilate links in it? I guess overall the question could be boiled down to, are affiliate sites hit because they have horrible content or because they have links to affiliate programs in them?
Affiliate Marketing | | bizzer0 -
Can anyone help me find the broadband ISP affiliate program I need?
I have a couple sites that deal with broadband internet access, and I'm looking to monetize beyond adsense. I'd like to sign up for some affiliate programs from internet service providers like Charter, CenturyLink, Comcast, etc. I found a couple affiliate programs but I am having no luck at all figuring out the programs that so many other websites are involved with. In about every case, the user is able to enter in their address and then a search is performed, returning programs available at their location. The programs I've found just do the traditional ad click-throughs for a possible commission. Can anyone enlighten me what kind of programs they are using?
Affiliate Marketing | | bizzer0 -
How much link juice passes through urls with affiliate id's?
hi we can get a valuable link with the desired anchor text from a news site. the destination url would be something like www.site.com/product. but in order to track conversions, our sales team would like to add an affiliate id to the url, so that it would look like this: www.site.com/product?sess_affiliate=ta how much link juice would a link to this affiliate url pass? would we be shooting our wad by linking to the ?sess-URL instead of the original URL?
Affiliate Marketing | | zeepartner0