Like you Vince, I went through the process of disavowing a number of links with Google and Bing and yes a week or so later I started to see a big rank dips. The links had been sending traffic that skewed my bounce rate and visitor page time rate, quite literally adding 1000's of visitors a month. Now after a few months the site has returned to its normal monthly stats and rankings are back to normal. But it took about 2-3 months for this to all work through.
I know how you feel 'Why did I do that? darn it' but I think like Tom says it's better to stick out and not remove the disavow file and focus on building good content for people to link too.
David