I can relate to this as well.
I remember when I was in high school, I suffered from anorexia and was very small. I think I weighed 100 pounds at most. I was in a dance class, and the teacher used me to demonstrate this kind of stunt lift we were going to do for a dance routine. As she set me down, she remarked (in front of the whole class) "Wow, you are way heavier than you look."
Since those days, I have gained and lost the same 100 pounds at least three times. I think that even though I am short, I tend to carry weight differently so it looks like I don't weigh as much as I do. Also, I suffer from a kind of reverse body dysmorphia, in which I always perceive myself as being smaller than I am.
I mentioned this on a forum once and someone congratulated me on my high self-esteem! It has nothing to do with self-esteem, because mine is in the toilet.
But it's almost like my body and mind only remember the days when I was a size 4-6 and those are the sizes I immediately grab when thrifting--even though I am a size 14 now.