Blocking external links in Robots.txt - need advice on Best Practice
-
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt.
In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to:
a/ leave it as is?
b/ allow crawling?
Thanks in advance.
-
I'm not sure how you're blocking Google from crawling external links in the robots.txt file--typically you only block them from crawling internal pages on your site.
If you're using a script, though, to track the clicks on external links and that script is running your site (and you're blocking that script in robot.txt), then that still should be fine. You may want to add a "nofollow" tag on those links, though, so you don't end up passing link "credit" or "link juice" to those affiliates (unless you want to do that?).
As far as external links go, though, it's typically okay (and expected) that you link out to other sites.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Person using expired domain and its links to drive traffic
Hi, I know about people using expired domains to drive juice to their primary site but what about people using AN expired domain as their primary site (totally changing that site into a trashy affiliate-marketing vehicle)? The site I'm looking at is thegunzone.com. It has, according to Semrush, almost 38K links. It used to be a legit 17-year-old firearms hobby site, and this is what it originally looked like: http://web.archive.org/web/20120213184627/http://thegunzone.com:80/ Here is its last page before it closed and the domain purchased by the affiliate marketer: http://web.archive.org/web/20170315084035/http://www.thegunzone.com/ It closed around February of 2017, and some affiliate marketer bought it and all its backlinks. However, all those backlinks, which were previously to various articles, are now directed back to those articles (which don't exist anymore) but the homepage, including Wikipedia links. Here's an example: https://en.wikipedia.org/wiki/Polygonal_rifling At the bottom, in the 7th Reference, there's a link to an article called " "Learning About Shooting . . ." but if you click on the original link, it just goes to thegunzone.com homepage. Again, the site's totally different. And there are just thousands of such backlinks to former articles that don't exist anymore but are redirected to this schlocky site's homepage (and it's passing its juice through too). My question is this: this cannot be kosher with Google backlinking policies, right? Is this prevalent on the internet? Why hasn't thegunzone.com been found out and its rankings penalized yet? And how do I report him? I see tons of other sites using this basic strategy too on search results with various hunting keywords. (Disclosure: I do own a hunting/firearms blog, but I don't do any backlinking at all.) Any help would be sincerely appreciated.
Affiliate Marketing | | HandyWoman1 -
Affiliates website, I need webpages linsk to study the topic
Kindly Provide me the best links where I can study about Affiliates websites, How to create them? there SEO? Content etc.
Affiliate Marketing | | bondhoward0 -
Considering Using External Domain Name To Build Separate Community
We are currently working with a client who sells classic Chevy muscle car parts: www.classicmuscle.com. They also own OldMuscleCars.com which is being redirected to the classic muscle site. Through our remarketing efforts in Adwords, we've noticed a growing number of affiliate blogs/forums/auto community sites (ex: classiccars.com) where customers are hanging out and seeing our banner ads. Our thought is to potentially create a new community site centered around muscle cars in general (not just chevy which is CM's specialty) that can be used in part as a lead gen tool (banners and text links would lead back to the classic muscle site) as well as a platform to explore other highly sought after vehicles (potentially expand outside of current chevy models or into other brands like ford). My questions is what are the implications of creating a site like this and are there any concerns (ranking or otherwise) with linking back to the classic muscle site? We're already ranked well for a majority of our current target keywords so link juice is really a secondary motive to lead gen. Best regards, Ryan
Affiliate Marketing | | Trone0 -
Where can I find best resources for Affiliate Marketing?
Any suggestion for Affiliate training, forums, seminars, etc. for newbie? I'm doing my research but would like to hear from pros. I'm experienced web developer and Adsense publisher, now want to step in the world of affiliate marketing.
Affiliate Marketing | | webplex0 -
Can linking out to a weak site harm my sites SEO?
We have an affiliate that wants us to link to his site to help him get started. His website looks okay and is in the same niche as us. However, he has PR 0 and no Page Trust. We could work in a natural looking link in a news article, but there is no way he could be considered any kind of "authority" link. Can linking to his site harm us?
Affiliate Marketing | | theLotter0 -
Does Google look at Domain Registrar owner information when counting links into a site?
Question: If the domain registrar owner info is the same on two websites, will Google/other search engines discredit any linking between the two sites when calculating page rank? We have two company sites. One is our main ecommerce domain which we have been linking into from information domains in the past. We recently cut ties with the company that hosted them for us and wish to host them ourselves to preserve the inbound traffic. We are worried, however, that the traffic will be discredited now that we own both domains. Thanks for looking!
Affiliate Marketing | | tennisexpress0 -
What is the negative effect of getting affiliate links to your domain?
Are the negative effects of affiliate links to your domain bigger than the positive effects? Is it better to use a subdomain for your affiliate campaigns? (Because of linkprofile no follow links or time on page effects?) Thanks for responce!
Affiliate Marketing | | ltom0 -
How to use different affiliate link based on geolocalisation in a Seo friendly way ?
Hi All, My website Sportytrader.com http://www.sportytrader.com/ works as an sport betting affiliate website on the french language markets. With sport betting legalisation in Europe, only people from France can play on french sport betting website. The same for Belgium or swiss. We have traffic from everywhere. So we need to geolocalize all our affiliate links to provide the right affiliate link to the right person in function of his location? What is the best way and the more seo friendly way to do that? Thanks a lot, Renaud
Affiliate Marketing | | jarnac0