Hey,
I was working with a site recently who's host suffered a number of issues over a few days, enough to receive a few warnings in GWT - and yes there was a hit to the rankings in the short term. However once their host got their game together the rankings did creep back up, although it took around a month of normal up-time for the ranking to largely recover.
As Greg says i'd definitely ensure you're on a good host, server downtime should be the last thing you need to be worrying about!
All the best,
Stuart