Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the danger of integrating different affiliate program widgets far under our core content?
We have pages with lots of unique content that are ranking very well in google.
Affiliate Marketing | | lcourse
I am now considering to add towards the end of the pages widgets from the booking.com affiliate program and also from the affiliate program of two other accommodtion sites. These affilliate widgets would be placed far beyond our core content in the bottom 15% of the page. Each widget will contain a large number of links to the different accomodation options that booking.com or others offer. In our context this information would be indeed very useful for our visitors. Shall I be concerned that our google rankings may suffer due to google perceiving our site as less quality (or google misinterpreting our intent to generate traffic to affiliate programs) ? If google rankings would be suffering, is this something we would normally see in a week (most of our pages are crawled daily by google) and then recover in a week when we would remove the widgets or would this generally be something that would affect the ranking over the course of a month or more? Shall I be concerned abou link dillution since my pages would now have a lot of links pointing to the affiliate sites, which could devalue my internal links? Any thoughts?0 -
What's the best way to go about making Duplicate Content Pages on my domain for affiliates?
Hello, I would like to make a bunch of duplicate pages of my site's Home Page, that way affiliates of mine can have a page of their own with links specialized to themselves littered throughout the page. What's the best practice going about this without jeopardizing domain authority from tons of duplicate content signals firing off?
Affiliate Marketing | | Benavest0 -
Blocking external links in Robots.txt - need advice on Best Practice
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt. In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to: a/ leave it as is? b/ allow crawling? Thanks in advance.
Affiliate Marketing | | Ben_Malkin_Develo0 -
Affiliate Program: Does anyone know how to set this up?
Hi Guys Does anyone know how I can make a affiliate programme on my website eg - I have a sales person in two of our local regions and I want to find a way I can track any sales made through them via my website so I can pay them their commission. Any advice around this, the website is wordpress! Cheers Justin
Affiliate Marketing | | edward-may0 -
How do you find Affiliate Links on your site that have not been nofollowed?
We've just signed up to an affiliate scheme because we were sending links to them because we thought their product was valuable to our users. So we now have to go through and nofollow all of these links over 100's of pages. Is there any way that do a crawl of the site to identify all links to a particular site and tell me what page they are on and whether they are nofollow/follow?
Affiliate Marketing | | Zippy-Bungle0 -
Affiliate Programs
Hi, I am wondering what if any affiliate management applications are being used out there. Anyone using Shareasale... I am looking into them.
Affiliate Marketing | | unikey0 -
In what ways would an affiliate site be okay with Google?
I know Google has slammed affiliate sites hard. But affiliate marketing can be a real business and had an idea for one, but I would like to know if ALL affiliate sites are looked at as evil in Google's eyes, or if it's just those with thin and duplicate content in the product descriptions. I know they want brands to come up ahead of affiliates of that brand, and that makes sense. If I sold Acme Widgets products, Acme Widgets itself should come up ahead of my site. But what if the site sold widgets in general, with the term widgets not a branded keyword? If all product descriptions are unique and well-written content, on a site that is high quality, would it still be downgraded by Google just because there are affilate links in it? I guess overall the question could be boiled down to, are affiliate sites hit because they have horrible content or because they have links to affiliate programs in them?
Affiliate Marketing | | bizzer0 -
How many affiliate links is considered too many?
Hi, Let's say you have great reviews for 50 products and some of these products do have affiliate links on review pages. And then you have user scores and you come up with a top 20 product list sorted by user scores. Now if you have the list of top 20 products on one page and all these products have an affiliate link (with nofollow relation and a 301 redirect) on the same page with only a couple of images and a summary of the review linked to the review pages, would this still be considered as what Google calls a "bridge page"? Would it be better to still generate the top 20 list but rather link to review pages only? (to avoid too many affiliate links on one single page).
Affiliate Marketing | | Gamer070