HIGHLY unlikely that your website got pandalized due to "duplicate video" content hosted on your server.
If the videos were embeds (youtube, etc) and the creator of the video has higher authority website and has the videos embedded too, then yes, your site might throw off 'thin' content trait but even then it shouldn't be enough to attract the Panda to its door.
If I were you, I'd look at all the pages for the text content they have and if any are duplicates or close to duplicates, and if I find any I would have original content written for each . I would also look the title and meta tags and check for duplicates also analyze see the # of backlinks to all the pages.If 1% of your websites pages have all the backlinks and rest have no links or social activity at all, that could throw off a thin content profile as well. I would also get lot of social signals (retweets, etc) - get those pages on Twitter and other social networks including Digg, Stumbleupon, etc to show Google that these pages don't simply exist for the sake of it but they are shared (liked) by people as well.
Take a look at other metrics as well - time on site, bounce rate, etc as they are getting more important.