I swim 3-4 times/week - and if you look at elite swimmers, they are very lean!

It *can* make you feel hungry immediately afterward, though ... something about your body trying to cool itself, etc. But I just make sure to have a good quality meal afterward and it's no problem. They also have water aerobics (but that will be mostly older people).