Hurting? NO!
Not Helping? This is not the case either.
Actually you are missing the opportunity to be smarter on links... If you are getting quality links with www then you should use the same URL as upon redirection (301) some of the link juice will be left.
As you said you made most of the links with-www then you should try redirecting all versions of URL to www-version so that you can earn the most out of it...