I guess my point really was that I think it is silly for someone to assume someone is vegetarian because they don't want a hamburger (or something else meaty). I know many people that don't eat beef. At one point, I was eating meat rarely and people would ask me if I was vegetarian. I would say no because I wasn't.
You can't out-exercise poor eating habits.