As Max said, from a ranking perspective, Time to First Byte seems to be the most important factor. The same author of that post offered some tips to improving time to first byte: http://moz.com/blog/improving-search-rank-by-optimizing-your-time-to-first-byte
Oftentimes, you simply have a lot of assets to load and it's difficult to cut anything back. In these cases, the order that things load becomes increasingly important for user experience (asynchronous java script, for example).
Regardless, doing everything you can to improve speed and checking with Google Page Speed Insights is usually the best advice. I've never, ever seen a website where improving speed performance didn't help with traffic metrics (wether rankings or engagement) so I believe it's an investment worth making.