First and foremost do NOT ditch your domain name. At best I'd say launch a new site in parallel and see how that goes but you'd basically be admitting defeat and starting over from scratch.
Second, buy some PPC. You're a chain so leverage that and get some PPC going to raise your traffic. PPC is also a great way to identify long-tail keywords you're missing. But don't just buy them. Take time and groom them daily. Pair them with Analytics. Suck the marrow out of them. Live them for the next week.
Third, realize you've set the bar really, REALLY high. You're chasing a generic keyword, and a massively competitive one at that (506M results in Google for the word "fabric"). In the post-Panda world, I'd say that's unrealistic. And why are you so obsessed with that one keyword? I've found single word terms are some of the poorest to go after. Lots of traffic, but major bounce rates and poor conversions (I would assume you want to sell, fabric, not just have people come look at it). Fabric could mean cloth, but it could also mean reality. Do you sell camo fabric? Google wants relevance above all and if people are bouncing out of your site, it will fall for that term. Generic terms, as a business model, are horrific.
What you need is to understand who your audience is and what they're looking for. Are they looking for bridal fabric? Maybe woodchipper macadamia fabric? Before you laugh, consider that maybe there is such a thing and a rich person who is buying a lot of it right now... somewhere else. I've found that owning a dozen smaller, niche terms that convert well beats the generic term any day. See #2 for how to find these juicy terms.
Then what? Content! You have a pretty site. Lots of things to interest. But you need to diversify. Your front page alone is way too busy. You have 6 banners stacked like one of those crazy subs at Subway. Each has value, but crammed in they have no value. And your deal of the day... uhm, they're sold out. I can find more in 19 hours??? Do you think I'll come back tomorrow to see what 3 fabrics sold out? Stagger them. Let me know there's another deal coming in 4 hours, 32 mins 23 secs, not 19 hours.
And your titles... ugh! Your main page is titled "Best Selling Fabric" but it's a lie! Thank you for visiting, Mario, but the best selling fabrics are on another page (just click past the Mushroom Kingdom patterns from the 1980s at the bottom). And what site did I visit? Hmm... the title doesn't tell me. Is this Beverly's? Fabric.com? Hobby Lobby? They're descriptive but they still miss the mark. You need your name at the end. Even Amazon does this (and they sell fabric too!).
Last but not least, fire your moving company. They've packed you in a box. Get out of there. Watch my swinging spoon tied to a string (I don't own a pocketwatch. Work with me). You are no longer an SEO. You are a marketer. You must convince me why I should buy your fabric. Show me the money fabric! Put it out! Make your site scream it. Right now your site is struggling to sell fabric to naked people (why naked people would be using Google is your problem to solve).