I believe you should leave the sitemap as it is for now. and Work on improving the quality of your website; The Pages, Design, Link Structure, URLs, Content... I mean everything
Improve the quality score of your pages and your rankings will surely rise again