It took many many months for me to say I was a vegetarian and really, I've only said that to a few people. (As I said before, I usually say I don't eat animal products) The issue really comes up in situations where people think they understand what vegetarian means. "Come over for dinner, I will cook something vegetarian for you" and then someone serves you fish, chicken or other animal products. Or even if you go to a restaurant and they have 'vegetarian' options that include fish.
Curious though that you say you were vegan but missed salmon, why couldn't you eat a mostly vegan diet and eat salmon? Are you classifying your current vegetarianism as pescatarian?
I think we are all responsible for what goes into our mouths but it can be a daunting task to eat out or even eat with friends sometimes. At most regular restaurants, I usually have 1 maybe 2 choices of things to eat off the menu. It is also refreshing when a restaurant understands vegetarian diets and tries to accommodate them. If accommodating started meaning 'fish products are ok', then it would be even more difficult.
I personally eat out at a vegan restaurant almost every week. Mostly because it is nice to go out and have lots of choices that are good and I don't have to worry about them not understanding what I'd want in my food and what I wouldn't.
You can't out-exercise poor eating habits.