This is a tricky one - you're more likely to come up for the smaller city if you're far from the city center, but it's not as likely to get as many searches.
I'd suggest using whatever address most commonly appears on the web. One thing Google really relies on is a consistent NAP - Name, Address and Phone Number. If your website says San Diego, and you have other local citations (directories, etc) that say San Diego, go with San Diego - that should help boost all your local rankings.
