Oh I'm sorry I clearly misunderstood the question.
I have not seen any studies or testing done on this, but I have to assume that they are ignored by spiders entirely. I certainly don't think they are more damaging than a 404 would be. A 404 tends to be ignored and only registered if a certain amount of time passes and the page is still not found. Google doesn't make it a habit to instantly remove URLs unless you ask them to.
At the very worst, the 403/404 error would de-index that particular URL but this should not affect the rankings of your other pages and your actual site. And I think it'll take at least a good 30 days before Google will stop crawling those. That said, it shouldn't be crawling them at all if there aren't any links pointing to them either internally or externally. And if there are links pointing to the pages in question, you should be redirecting them via 301. That is of course if they are links you want.
Hope this was more helpful.