Hi Stephane,
I would suspect that those links are a problem. If they make up a majority of your link profile then it looks "unnatural" to Google. Usually, your highest frequency of Anchor Text should be your brand name or home page. Your best bet is to use Open Site Explorer and check your Anchor Text distribution. If this link appears in the footer or sidebar you are raising another red flag.
After doing some research your best bet would be to ask that webmaster if they can add the "no follow" attribute to a majority of the links. You might want to find strong pages on their site to ask for an in text link in order to keep some of the link juice. These links are generally editorially given and do not raise red flags.
Hope that helps!