I would normally say "Alexa doesn't matter so don't worry about it" but it turns out a lot of advertisers and domain name buyers do consider it when determining the value of a site so there are definitely some benefits to having a decent Alexa ranking.
Anyways, to answer your question, in order to improve your Alexa ranking (without blatantly cheating), you basically need to increase the number of visitors to your website that have the Alexa toolbar installed. Here are some suggestions -
1. Most anti-spyware programs nowadays flag the Alexa toolbar as malware and remove it from the computer, so this means most Alexa toolbar users are not as computer savvy and don't even know how they got the toolbar installed in the first place, let alone how to remove it. Take this into consideration when creating content/targeting keywords specifically for this kind of audience (ie, likely the older generation).
2. Do keyword research to see what kind of searches are being made for "Alexa" related keywords such as "Alexa toolbar uninstall" or "remove Alexa toolbar." This is a dead giveaway that these users have the toolbar installed and are looking to remove it. You can get them to your site by ranking organically for those phrases or if you want these visitors quickly, start up an AdWords campaign and bid on "Alexa toolbar removal" related keywords. Sit back and watch your Alexa ranking take off.
3. My final suggestion is my least favorite since it's essentially peddling spyware to your user base but it's still effective. Offer your users the option to download and install the Alexa toolbar from your website in exchange for some kind of reward. Again, I wouldn't suggest going this route since you risk the chance that your users will abandon your site forever but nonetheless it's still an option.