I would suggest the kind of linking and marketing you are doing away from your seasonal times. If you are still link building and trying to push rankings during your 'off peak' season I don't see why you should nosedive.
Look at ways in which you can use the off peak to really push your marketing forward. For example creating content and just generally being active throughout the year and still trying to build your back link portfolio.
Obviously you want to market toward the time when you get the most business, but 'off season' can be just as important.
It might be that more people are linking to you and talking about you during your seasonal time and Google is seeing this and ranking you accordingly, but when that stuff dies away and not much more comes through your competitors begin to rank above you due to the kind of SEM they are doing.
Hope that helps in some way. Any more information could help give a better analysis.