Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What impact do affiliate links have on SEO
We commonly reference engineering standards on our site (because half our products are known and searched for by those standard names) and have just signed up as an affiliate to an organisation that sell the standards. Which seems a nice match We're going through and making sure all are no follow now because that seems the general advice - but should we be doing anything else? Yoast says you should mask affiliate links - https://yoast.com/affiliate-links-and-seo/ - but that was from 2010. Matt Cutts is fairly clear on nonfollow - http://www.webpronews.com/heres-what-googles-matt-cutts-says-about-affiliate-links-and-nofollow-2012-06 - but not on whether it affects SERPS Does it? And what is the best way to handle it? This is a typical example of a page where we would have an affiliate link or two. http://www.oakleysteel.co.uk/fake-mill-certificates-en10168
Affiliate Marketing | | Zippy-Bungle0 -
Which is the Best Affiliate Marketing Platform in Australia?
We have 1 eCommerce website (www.bannerbuzz.com.au) for printing products like banners, signs, banner stands, teardrop flags and more... We want to start affiliate marketing for our products with CPA model. So, Do anyone have a good experience with Australian affiliate marketing platform. We are having shareasale account for our USA based website, But Shareasale is not that much aggressive in Australia. So we want to finalize 1 platform for affiliate marketing which can have good strength of advertisers and reputation. Thanks,
Affiliate Marketing | | CommercePundit0 -
Low value link building to sitemap.xml
During some competitive research recently I discovered one of my clients competitors sites had an interesting backlink profile. Looking at the top-pages report in Open Site Explorer the home page was the #1 page (as you'd expect) with 2.5k links from about 500 linking root domains The second page was the sitemap.xml (~1.5k links, 400 linking root domains) and the third was their /feed page (again, ~1.5k links, 350 linking root domains). Links to these two pages aren't something that would happen naturally (particularly the sitemap.xml). There's a whole load of evidence for nasty low quality link building such as over-optimised keyword rich anchor text, comment spam, and even some blog/article based link networks. It's a pretty nasty niche with lots of cut-throat affiliate marketing. My guess here is that someone may have made a mistake using an automated link building too, but I'd be interested in what you might think? Have you seen this before? (Sorry, I can't reveal the domains in question as I'm bound by an NDA.)
Affiliate Marketing | | DougRoberts0 -
Tricky Situation. Need the communities help on this. Need Creative ideas. HELP! !!!!
Hello Moz Community . I currently work with an e-commerce company who is the health and beauty sector. The company is about 7 years old and been online for only 2 years. Income comes from 90% from wholesales/retail stores ( owned by franchisees ) . The entire worldwide chains generate about 300 to 400 sales per day with an average sale price of $125 per sale. The problem i have had for the last year is the boosting online sales. ( we are a very unknown brand ). I have missed about 40% of my target goals and its becoming frustrating. There is my current dilemma and i need some creative ideas. The wholesales have a hold on the current market and sale prices of the products. We need to make sure we ok the prices with all the wholesale franchisees before doing anything. We cannot do promotion upfront on the website because the wholesales work with a discount tactic to sell products in the store. Although it is unorthodox , it works for them. My Dilemma is we have a potential 300 to 400 customers per day i can convert to online customers, but i cannot market to them in the store what so ever. I cannot ask the wholesale franchisees to get their email address because they will screen from the mountain tops that we are steeling customers from them . My question to you guys is , what creative way can i grab customers attentions and of course Email - Addresses to potentially switch them over to online customers. Just a reminder i cannot Do anything in the stores, but however i can do anything i want with the product. And yes we have our website address on the boxes and products and we even have a life time guarantee on products when they sign up online, But we convert customers at a measly 3.5% with that gimmick . I had thought about putting a little tag on the products or a little booklet inside the packaged boxes and basically invite the customers to order a free sample set of our products. I wanted to create a landing page and then have them input their information. Then i thought about magnifying the procedure by stating " invite your friends to receive Free samples of our products. " and having them input their friend and family email addresses. I do feel its a solid idea i have but i wanted help by asking some of the communities creative mind. If you have any questions about anything please feel free to ask and thank you in advance for all your help. Best Wishes, Hampig M.
Affiliate Marketing | | BizDetox1 -
Merchant´s data feed for affiliates is the same content as their own website...
Hi Some advice appreciated. Started working on a site and found out that they are giving their unique content to their affiliates (an XML feed so appearing on another domain). In this case, if they want to provide the data like that, how can we protect ourselves? Should we use author tags in our html, is that necessary? Is there any fix other than "stop doing that and give them different content"? Thanks
Affiliate Marketing | | xoffie0 -
Where to look for relevant links
I and many others are targeting keywords such as "free iphone" or "free ps3" as part of incentive marketing schemes that offer rewards for referring new users to businesses. These often seem too good to be true but actually do work (if you don't get caught up in the spam sites). The thing is, what sort of link building should I be looking at once I have my site showing the users how the opportunity works and that it is a legitimate site. I don't want to have free iphone links placed where people may think I'm scamming them, so should I look at press releases, creating infographics or what? I look forward to hearing some of your ideas...
Affiliate Marketing | | GrassRootsSEO0 -
How should I deal with the affiliate URLs we create through our affiliate program?
Our website offers an affiliate program and every affiliate gets a link http://www.ourwebsite.com/?affiliateID. Because we get many such links we use redirects to avoid duplicate content. Unfortunately we have 302 redirects set up and I want to make them 301. Many of our affiliates build backlinks using their http://www.ourwebsite.com/?affiliateID as the URL. If such links were redirected through a 301, would they pass any juice? Can our site get penalized by search engines for having backlinks to URLs that contain affiliate IDs, URLs that are 301 redirected to our homepage lets say?
Affiliate Marketing | | nick.leech0 -
Is setting up affiliate marketing on website worth it for a small businesses?
Is setting up affiliate marketing on website worth it for a small businesses?
Affiliate Marketing | | seoukrsf0