Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Person using expired domain and its links to drive traffic
Hi, I know about people using expired domains to drive juice to their primary site but what about people using AN expired domain as their primary site (totally changing that site into a trashy affiliate-marketing vehicle)? The site I'm looking at is thegunzone.com. It has, according to Semrush, almost 38K links. It used to be a legit 17-year-old firearms hobby site, and this is what it originally looked like: http://web.archive.org/web/20120213184627/http://thegunzone.com:80/ Here is its last page before it closed and the domain purchased by the affiliate marketer: http://web.archive.org/web/20170315084035/http://www.thegunzone.com/ It closed around February of 2017, and some affiliate marketer bought it and all its backlinks. However, all those backlinks, which were previously to various articles, are now directed back to those articles (which don't exist anymore) but the homepage, including Wikipedia links. Here's an example: https://en.wikipedia.org/wiki/Polygonal_rifling At the bottom, in the 7th Reference, there's a link to an article called " "Learning About Shooting . . ." but if you click on the original link, it just goes to thegunzone.com homepage. Again, the site's totally different. And there are just thousands of such backlinks to former articles that don't exist anymore but are redirected to this schlocky site's homepage (and it's passing its juice through too). My question is this: this cannot be kosher with Google backlinking policies, right? Is this prevalent on the internet? Why hasn't thegunzone.com been found out and its rankings penalized yet? And how do I report him? I see tons of other sites using this basic strategy too on search results with various hunting keywords. (Disclosure: I do own a hunting/firearms blog, but I don't do any backlinking at all.) Any help would be sincerely appreciated.
Affiliate Marketing | | HandyWoman1 -
Affiliate website redirection concern
First of all i want to say hi to everyone here since im a new guy! The issue: My website is a price comparison website. I create products and inside these products i list several shops with their priced offers from the lowest to the highest. When a customer clicks the "shop now" button to go to the desired shop a new window pops up telling the user that he is being redirected to that shop. Check the website here and click "shop now" to understand what i mean: https://www.pccdkeys.com/product/buy-world-of-warcraft-legion-cd-key-for-battlenet/ This pop up page is a custom .php called redirect.php. In order to create a unique url for each shop, we add parameters in every URL like this: /redirect.php?id=5030917189616&merchant=CDKEYS&game=4117. The redirect.php right now is under this setting: custom header 301 without destination URL (which probably is wrong) it is noindex, follow in meta robots the parameters: **id, merchant, game **are set to NO URL's in URL parameters in webmaster tools. If i remove the 301 redirect from the header it will lead to 200 OK but then its a soft 404 error because the page is small and for sure i dont want 8000 of soft 404 errors because google phantom will destroy me. Another setting was: No custom header so it was 200 OK noindex, nofollow (which was for extra protection from the disallow) disallowed in robots.txt Thi setting kept giving soft 404 errors in audits. I also had it with this setting: custom header 301 without destination URL meta robots in index, follow without disallow in robots This setting gave me Not followed errors in google webmaster tools. The question: Which is the proper setting for this php page? Maybe a combination of the above? Thanks in advance and sorry for the long post.
Affiliate Marketing | | dos06590 -
Advise about Social Sharing between two loosely affiliated website - (retweets/fb cross sharing)
So basically, I have two websites. One being my Primary website and another which is very Niche/Seasonal. Now the Niche website received a penalty, and was later re-included however google traffic has been extremely limited since being reindexed. (bing never had a problem, still top ranked) I don't directly link back and forth between the two websites because its not worth hurting my primary websites ranking. I'm sure google knows I own both anyway, as they are on the same webmaster tools but the Niche site was kicked for spammy link building & its been an endless process removing them all. Anyway, with that being my concern. Would it be acceptable to cross share facebook posts, retweet and do things like that on occasion? My primary reason for asking is because the Niche website has a very large social following, so I would like to share post/tweets from my Primary site when its relevant and vice versa. Appreciate any suggestions/comments. Thanks!
Affiliate Marketing | | Southbay_Carnivorous_Plants0 -
Tweeting affiliate links...?
I work in the music business & am published music producer. I'm an authority in my niche. Yada yada. I have a website that sells products to other music producers. I recently set up an affiliate partnership with someone who is also in the music business but on the management side. She is not an authority to my customers as far as music making goes. But she does have a huge following of people who are EXACTLY who I am targeting. She has been tweeting my affiliate links. The traffic has been great, but the purchase response has not been. I've had much better responses from reviewers and other music producers who have tweeted my links. I was just wondering if anyone had any tips for the most effective ways of tweeting affiliate links? Thanks!
Affiliate Marketing | | PedroAndJobu0 -
Affiliate URLs Indexed in Google
We have an active in-house affiliate program which has a create-a-link function where affiliate, as in most affiliate programs, can build links back to our content to produce sales. The problem that I am seeing is that some of the affiliate URL versions are being indexed in Google rather than the original page. For instance.. http://www.ourdomain.com/page_start.aspx?affnum=F025212&Start_Page=RaceCar&referrer=createlink is outranking... http://www.ourdomain.com/racecar This presents 2 issues for us. First, this presents duplicate content issues obviously. Second, we pay our affiliates a portion of the sales so, in order to maximize profitability, we'd like the indexed pages to be the original version. What's the best way to handle this? I want to make sure that affiliates continue to get credit for links from their sites but search engine links should go to the original versions.
Affiliate Marketing | | ATIseo0 -
Does having affiliates hurt seo?
one of my clients has a TON of affiliates out there building links back to his site all with the ?ref=AFFILIATENAME tracking string does google understand this and not count these links or is there a way to nofollow all links with this tracking string? just wondering how big companies rank well and still have thousands of affiliates building links on all kinds of properties using all kinds of methods thanks
Affiliate Marketing | | Ezpro90 -
Linking from High Authority Sites
Dear Mozzes, We're running a property portal for real estate classifieds which receives around 300.000 visits monthly. PA 47 DA 37 Mozrank 5,3 MozTrust 5,8 PR 4 Apart from our real estate classifieds we are producing proprietary trends and statistic, a rare commodity in the country we operate. Luckily we are about to enter into a partnership with one of the leading newspaper portals in the Country, appreciating PAs in the 80-90s. In this coming partnership we will provide: 1. Market statistics, which we can imagine as a chart followed by our logo and a "Powered By", which shall link to our site. 2. An in-framed part of our site in their portal where we list properties for sale, that will carry affiliate strings. In order to for us to get the maximum link value of this high authority site, how should we build the integration and linking? We've looked at similar examples here http://realestate.money.cnn.com/ here http://realestate.money.cnn.com/NY/New_York/ or maybe like the Simply Hired logo here: http://money.cnn.com/ Advices here would be super appreciated. Thanks Eric
Affiliate Marketing | | PropertyPortal0 -
Affiliate links and Panda
Hello all, my question relates to the recent panda changes and the subsequent impact on pages with affiliate links. I have a hub of pages on my website talking about parcel delivery. Each page has unique, quality content and up until about 3 weeks ago they were starting to achieve strong rankings. We then decided to add some affiliate links (just 3) on each of the 5 parcel related pages we have. The rankings for each of these pages have tanked completely; rather than top 20, we're now outside of the top 100 for all of them. When we noticed this we masked the link, changing it to domain.com/index/affiliate; we also no followed the link. Alas this has had minimal effect in restoring the rankings. What i'm now thinking of doing is removing the affiliate links from the parcel landing pages altogether and instead providing users with a no followed link to another page within the site upon which we would have the affiliate links. This page would also be no indexed as well. My question is: Is this a good idea and if so will it restore my previous rankings over time, or have those pages (as a result of the affiliate links) been permanently down graded?? Any assistance would be gratefully received!
Affiliate Marketing | | danielparry0