Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Person using expired domain and its links to drive traffic
Hi, I know about people using expired domains to drive juice to their primary site but what about people using AN expired domain as their primary site (totally changing that site into a trashy affiliate-marketing vehicle)? The site I'm looking at is thegunzone.com. It has, according to Semrush, almost 38K links. It used to be a legit 17-year-old firearms hobby site, and this is what it originally looked like: http://web.archive.org/web/20120213184627/http://thegunzone.com:80/ Here is its last page before it closed and the domain purchased by the affiliate marketer: http://web.archive.org/web/20170315084035/http://www.thegunzone.com/ It closed around February of 2017, and some affiliate marketer bought it and all its backlinks. However, all those backlinks, which were previously to various articles, are now directed back to those articles (which don't exist anymore) but the homepage, including Wikipedia links. Here's an example: https://en.wikipedia.org/wiki/Polygonal_rifling At the bottom, in the 7th Reference, there's a link to an article called " "Learning About Shooting . . ." but if you click on the original link, it just goes to thegunzone.com homepage. Again, the site's totally different. And there are just thousands of such backlinks to former articles that don't exist anymore but are redirected to this schlocky site's homepage (and it's passing its juice through too). My question is this: this cannot be kosher with Google backlinking policies, right? Is this prevalent on the internet? Why hasn't thegunzone.com been found out and its rankings penalized yet? And how do I report him? I see tons of other sites using this basic strategy too on search results with various hunting keywords. (Disclosure: I do own a hunting/firearms blog, but I don't do any backlinking at all.) Any help would be sincerely appreciated.
Affiliate Marketing | | HandyWoman1 -
Blocking external links in Robots.txt - need advice on Best Practice
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt. In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to: a/ leave it as is? b/ allow crawling? Thanks in advance.
Affiliate Marketing | | Ben_Malkin_Develo0 -
Any Ideas for SEO opportunities at an electronics convention (CES)?
Hello All, I'm currently helping someone that sells a unique 35mm slide scanner with his digital presence. For a quick explanation, it scans much faster than it's competition and targets photography/scanning businesses. The owner will have part of a booth at CES (large electronics convention) to display the product, and will also have time to network with others within this industry and many others. At last years convention, there was a good in-person response and a few articles written about the product, but that is about it. I would like to send him to the show with a more digital mindset, so I am wondering if anyone has any ideas for link-bait, promotions, or really anything that will help the website rank higher/get more traffic. We have tons of content ideas and other ideas that are "online" to drive traffic, but when it comes to a in-person event I am drawing a big old blank (but I feel there must be some opportunities that I am missing). If you have any ideas and would like to share, it would be greatly appreciated!!! Thanks! Eric
Affiliate Marketing | | EHuhn30 -
Does having affiliates hurt seo?
one of my clients has a TON of affiliates out there building links back to his site all with the ?ref=AFFILIATENAME tracking string does google understand this and not count these links or is there a way to nofollow all links with this tracking string? just wondering how big companies rank well and still have thousands of affiliates building links on all kinds of properties using all kinds of methods thanks
Affiliate Marketing | | Ezpro90 -
Duplicate tracking of Affiliate and Organic orders
I started diving into the over lap between Organic and affiliate orders and can see that on average about 10% of my organic orders get tracked as both affiliate and organic (meaning i am paying affiliate commissions on them). I want to be able to better account for the different types of order and where they are coming from. My question is simply if anyone has any ideas as to how better to figure out if those orders should be canceled on the affiliate end or if there is a better way in analytics to see where the order is actually getting attributed to through some type of goal or advanced set up or something (the download of orders from organic that I compared to the affiliate ones was taken from Ecommerce > Transactions > Medium (Organic) then I compared that to the order numbers from the affiliate company I am using). Hope this makes sense and thanks for any information that can help with this!
Affiliate Marketing | | Gordian0 -
Linking from High Authority Sites
Dear Mozzes, We're running a property portal for real estate classifieds which receives around 300.000 visits monthly. PA 47 DA 37 Mozrank 5,3 MozTrust 5,8 PR 4 Apart from our real estate classifieds we are producing proprietary trends and statistic, a rare commodity in the country we operate. Luckily we are about to enter into a partnership with one of the leading newspaper portals in the Country, appreciating PAs in the 80-90s. In this coming partnership we will provide: 1. Market statistics, which we can imagine as a chart followed by our logo and a "Powered By", which shall link to our site. 2. An in-framed part of our site in their portal where we list properties for sale, that will carry affiliate strings. In order to for us to get the maximum link value of this high authority site, how should we build the integration and linking? We've looked at similar examples here http://realestate.money.cnn.com/ here http://realestate.money.cnn.com/NY/New_York/ or maybe like the Simply Hired logo here: http://money.cnn.com/ Advices here would be super appreciated. Thanks Eric
Affiliate Marketing | | PropertyPortal0 -
Affilate Links
Hi I’m looking for confirmation of my worries about affiliate links to a website I'm working for and hopefully get some ideas of how to solve the problem of having affiliates without losing the strength of links from affiliate web sites. Poplidays.com has an affiliate program with an independant affiliation platform, but it also has a couple of partners who link directly to Poplidays.com from their sites. Poplidays wants to reward the partner sites in the same way as it does affiliates: with a percentage of sales generated by visitors. However as their SEO I don’t want to lose these links. They are from established, quality sites in the same subject area and, in my opinion, have a lot of SEO value. Affiliate links are 302 redirects from urls on the affiliation platform’s domain; something like http://poplidays.affiliation.com/?clientid=123 to www.poplidays.com Can you confirm that replacing a direct link to www.poplidays.com with a 302 redirect of this type will be a bad for Poplidays rankings? Any proof of this from a reliable source that I can quote? The affiliation platform is insisting that there is no negative effect and that no one has ever made a remark like mine before. Any good ideas of how to resolve the problem of rewarding linkers without spoiling the links for SEO? Bearing in mind that Poplidays want to be able to reward sites for visitors who may buy on a later interaction with the site (up to 30-days) and want an automatic report so that the partner can follow his commissions. I’ve tried implementing something through Google Analytics but it wasn’t perfect. Your in house affiliation system is working with links like http://go.seomoz.org/aff_c?offer_id=1&aff_id=1624&url_id=1&file_id=24 that 302 to http://www.seomoz.org/features?affid=1624&txnid=10274208218121120514 Is this a better / perfect solution ? Compared to a direct link to http://www.seomoz.org/features ? Thanks - Neil
Affiliate Marketing | | CarmenImmobilier0 -
Link juice from my affiliates to my site
I'm setting up an affiliate program through 1shoppingcart. Even though I run 3 different websites, the way their program works is that I have to have 1 domain name that my affiliates get as their affiliate link, with their unique id attached at the end. I can give them multiple links that will redirect to various places on my 3 websites, but those original links that they put on their site will all be to just 1 domain. I'm wondering if I still get link juice to my sites even though the links are obviously being redirected? Does it depend on the redirection method? I was just thinking that getting hundreds of affiliates who run related sites could be a very good seo boost if they're all linking to my site, but I feel like it's going to be wasted because of this redirection thing. Hope this made sense. Thanks very much,
Affiliate Marketing | | philraymond
Phil0