I think it was meant as a compliment, but I sympathize with how you feel. I am also a very conservative dresser, but I have a very "curvy" body, even with weight gain.
Once I pulled on a pair of black jeans which were tight, but I thought I had a longish top on (which was modest and loose). As I walked through a mall to get a pair of shoes, a 20-something man cupped his hands around his mouth, and yelled "ATTENTION: A**." A bunch of guys sort of grouped around me as I walked by...I was mortified to realize it was my A** he was alerting them to.
I have no idea if that was a compliment or not! It took me a while to go shopping there again. (And I can't believe I am sharing this story....I thought it would follow me to the grave!)