I think your physical appearance may influence a man's opinion of you because of what it may suggest about you, apart from the simple issue of attractiveness. Whether the impression given is accurate or not, it may be a factor.
For example, my fiance is very lean and fit. I asked him once what attracted him to me and he said "you looked like you could keep up with me". He was partially joking, but the truth is that he loves to mountain bike and snowboard and he was hoping to find someone who would do those things with him. I have complete confidence that he would still love me if I gained substantial weight. But his initial reaction to me was that I looked like someone who was compatible with him. I am fine with that - it does not seem superficial or shallow, especially as I was thinking something similar (I've always wanted a boyfriend who would go on long runs with me). I think studies have shown that people look for someone who looks like them, above all other factors. My theory is that it reflects our search for a compatible mate.
My .02, FWIW.