Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Blocking external links in Robots.txt - need advice on Best Practice
-
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt.
In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to:
a/ leave it as is?
b/ allow crawling?
Thanks in advance.
-
I'm not sure how you're blocking Google from crawling external links in the robots.txt file--typically you only block them from crawling internal pages on your site.
If you're using a script, though, to track the clicks on external links and that script is running your site (and you're blocking that script in robot.txt), then that still should be fine. You may want to add a "nofollow" tag on those links, though, so you don't end up passing link "credit" or "link juice" to those affiliates (unless you want to do that?).
As far as external links go, though, it's typically okay (and expected) that you link out to other sites.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My affiliate links are being crawled by Google. Is this a bad thing? Also, should I add rel: nofollow to affiliate links?
Hi All, I have an issue on Google Search Console. It seems all my affiliate links are being crawled. Is this a bad thing? The reason I found out is that all but one link was flagged with a warming. It goes on to say that it was blocked by robots.txt. The strange thing is that I've not set any links to be blocked in robots.txt. Also, should affiliate links be nofollow? I'm asking because I was recently sent this article by Yoast on how to properly cloak my affiliate links without using a plugin. https://yoast.com/cloak-affiliate-links/ In the method, they apply the nofollow and noindex rules to the links. I'm just a little confused on how to go about affiliate links the SEO friendly way.
Affiliate Marketing | | nomad_blogger0 -
What's the best way to go about making Duplicate Content Pages on my domain for affiliates?
Hello, I would like to make a bunch of duplicate pages of my site's Home Page, that way affiliates of mine can have a page of their own with links specialized to themselves littered throughout the page. What's the best practice going about this without jeopardizing domain authority from tons of duplicate content signals firing off?
Affiliate Marketing | | Benavest0 -
Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
Hey all, I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something? Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis. Bi75jku
Affiliate Marketing | | BradsDeals0 -
How do you find Affiliate Links on your site that have not been nofollowed?
We've just signed up to an affiliate scheme because we were sending links to them because we thought their product was valuable to our users. So we now have to go through and nofollow all of these links over 100's of pages. Is there any way that do a crawl of the site to identify all links to a particular site and tell me what page they are on and whether they are nofollow/follow?
Affiliate Marketing | | Zippy-Bungle0 -
Affiliate program links
Hello, I've question regarding affiliate partner generated links.Company www.example.com has a affiliate program. Partner www.example2.com links to the company site with affiliate link www.example.com/?affid=xxxx in banner, article etc. As I understand these type of links could hurt our Company site, so I asked partners to use rel="nofollow" in all links where they link to us usign ?affid=xxxx in url. To be 100% sure that I did the right thing I decided to post here my question. Could these type of links hurt our Company site? Regards,
Affiliate Marketing | | juris_l
Juris0 -
Does Google look at Domain Registrar owner information when counting links into a site?
Question: If the domain registrar owner info is the same on two websites, will Google/other search engines discredit any linking between the two sites when calculating page rank? We have two company sites. One is our main ecommerce domain which we have been linking into from information domains in the past. We recently cut ties with the company that hosted them for us and wish to host them ourselves to preserve the inbound traffic. We are worried, however, that the traffic will be discredited now that we own both domains. Thanks for looking!
Affiliate Marketing | | tennisexpress0 -
Can anyone help me find the broadband ISP affiliate program I need?
I have a couple sites that deal with broadband internet access, and I'm looking to monetize beyond adsense. I'd like to sign up for some affiliate programs from internet service providers like Charter, CenturyLink, Comcast, etc. I found a couple affiliate programs but I am having no luck at all figuring out the programs that so many other websites are involved with. In about every case, the user is able to enter in their address and then a search is performed, returning programs available at their location. The programs I've found just do the traditional ad click-throughs for a possible commission. Can anyone enlighten me what kind of programs they are using?
Affiliate Marketing | | bizzer0 -
How many affiliate links is considered too many?
Hi, Let's say you have great reviews for 50 products and some of these products do have affiliate links on review pages. And then you have user scores and you come up with a top 20 product list sorted by user scores. Now if you have the list of top 20 products on one page and all these products have an affiliate link (with nofollow relation and a 301 redirect) on the same page with only a couple of images and a summary of the review linked to the review pages, would this still be considered as what Google calls a "bridge page"? Would it be better to still generate the top 20 list but rather link to review pages only? (to avoid too many affiliate links on one single page).
Affiliate Marketing | | Gamer070