Blocking external links in Robots.txt - need advice on Best Practice
-
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt.
In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to:
a/ leave it as is?
b/ allow crawling?
Thanks in advance.
-
I'm not sure how you're blocking Google from crawling external links in the robots.txt file--typically you only block them from crawling internal pages on your site.
If you're using a script, though, to track the clicks on external links and that script is running your site (and you're blocking that script in robot.txt), then that still should be fine. You may want to add a "nofollow" tag on those links, though, so you don't end up passing link "credit" or "link juice" to those affiliates (unless you want to do that?).
As far as external links go, though, it's typically okay (and expected) that you link out to other sites.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Person using expired domain and its links to drive traffic
Hi, I know about people using expired domains to drive juice to their primary site but what about people using AN expired domain as their primary site (totally changing that site into a trashy affiliate-marketing vehicle)? The site I'm looking at is thegunzone.com. It has, according to Semrush, almost 38K links. It used to be a legit 17-year-old firearms hobby site, and this is what it originally looked like: http://web.archive.org/web/20120213184627/http://thegunzone.com:80/ Here is its last page before it closed and the domain purchased by the affiliate marketer: http://web.archive.org/web/20170315084035/http://www.thegunzone.com/ It closed around February of 2017, and some affiliate marketer bought it and all its backlinks. However, all those backlinks, which were previously to various articles, are now directed back to those articles (which don't exist anymore) but the homepage, including Wikipedia links. Here's an example: https://en.wikipedia.org/wiki/Polygonal_rifling At the bottom, in the 7th Reference, there's a link to an article called " "Learning About Shooting . . ." but if you click on the original link, it just goes to thegunzone.com homepage. Again, the site's totally different. And there are just thousands of such backlinks to former articles that don't exist anymore but are redirected to this schlocky site's homepage (and it's passing its juice through too). My question is this: this cannot be kosher with Google backlinking policies, right? Is this prevalent on the internet? Why hasn't thegunzone.com been found out and its rankings penalized yet? And how do I report him? I see tons of other sites using this basic strategy too on search results with various hunting keywords. (Disclosure: I do own a hunting/firearms blog, but I don't do any backlinking at all.) Any help would be sincerely appreciated.
Affiliate Marketing | | HandyWoman1 -
Affiliate program links
Hello, I've question regarding affiliate partner generated links.Company www.example.com has a affiliate program. Partner www.example2.com links to the company site with affiliate link www.example.com/?affid=xxxx in banner, article etc. As I understand these type of links could hurt our Company site, so I asked partners to use rel="nofollow" in all links where they link to us usign ?affid=xxxx in url. To be 100% sure that I did the right thing I decided to post here my question. Could these type of links hurt our Company site? Regards,
Affiliate Marketing | | juris_l
Juris0 -
Where can I find best resources for Affiliate Marketing?
Any suggestion for Affiliate training, forums, seminars, etc. for newbie? I'm doing my research but would like to hear from pros. I'm experienced web developer and Adsense publisher, now want to step in the world of affiliate marketing.
Affiliate Marketing | | webplex0 -
What is the best way to boost no. of quality sites linking in?
Analysing my site ranking compared to competitors, I see that our weakness is getting others to share our content and link into it via their own blogs and social media - although I would say our social media activity is pretty good. We have also worked on getting listed free on relevant sites related to our product area (learn spanish in spain) in order to increase sites linking but this is still minute compared to our 2 main competitors. (As a note: paid listings are not totally justifiable for us at this point in time but perhaps in the future we will expand on this). Am I missing something here?
Affiliate Marketing | | sparkit0 -
Need help about facebook fanepage!
I have a fanepage on facebook , some time have other fanpage go to my fanepage and spam..., now i want protect for they can't look at my fanpage :(( Who can help me thank very much! [link removed by staff]
Affiliate Marketing | | ITVSEO.COM0 -
I need help setting a campain
I am setting a campaign and I put the first page as a main route. Sample www.sample.com On this page people need click on a enter button because it is and adult page. After that everything redirects to www.sample.com/home.php Where you find all the links on the web page such as about, blogs, contact, promotions and so on. The problem that I see it's that I already set up the campaign but detects only one page. How can I see the errors of missing meta tags of the other inside pages. Should I set up the campaign and in my domain shall need to be www.sample.com/home.php instead of www.sample.com any wise advise on this?
Affiliate Marketing | | lnietob0 -
How much SEO benefit am I getting from 'affiliate' links?
Hi all, We run lots of different campaigns with affiliates and as such they have links on their websites pointing back to ours. I was keen to know how much 'link-juice' these kinds of links are passing back to our websites or how much SEO benefit we are getting from them. The links have "?=affiliate" in them. Any thoughts would be greatly appreciated!! Many thanks! Rob
Affiliate Marketing | | RobertHill0 -
Affilate links - Google Panda 2.1
With the recent update from Google Panda would links from affiliate Coupon Sites now have an negative impact in Organic Rankings or site value? A reason for this question is because many Coupon Sites usually have poor content and a poor quality score. Thanks
Affiliate Marketing | | Tonyd230