I can completely sympathize. Even at my current weight, I refuse to buy a swimsuit that doesn't have a little skirt (it's finally possible to find something fashionable where it doesn't look like you're wearing your grandmother's swimdress). It's not so much about having the right body to wear a swimsuit as finding the most flattering suit for your features.
Just two weeks ago, I went out in public wearing...SHORTS!

The last time I did that was 6 years ago. No matter how hot the temperature got, I always wore pants or maybe a knee length skirt. Some people may be able to say "it doesn't matter what other people think;" I understand the sentiment, but ultimately you are the one who needs to feel comfortable. It may take some extensive searching and $$$, but it will be well worth it to find the proper garment.