Swimming is a fantastic workout. It works almost every muscle group in your body and is great for cardio endurance because you are forced to regulate your breathing while you do it. But it is like any other exercise, if you don't push yourself, it won't be effective.
Body fat is an advantage in swimming because fat makes you more buoyant. So the swimmers that have a tendency to maintain a little more body fat are probably more successful. But this doesn't mean that swimming isn't a good exercise.
I try to swim at least once a week. I swim 2400 yards (just under 1.4 miles) in an hour. I make sure to include 800 yards (about 15 minutes) of intervals, where I swim one lap as fast as I can and then swim a recovery lap. I also try to swim 200 yards at a time without stopping (including flip turns if I'm swimming freestyle). This workout has served me really well. During the weeks that I swim, I notice a marked improvement in my cardio endurance (I can actually set the treadmill to a faster setting and run at that setting for longer). Also, the swim workout gets easier every time I do it.
I'm not sure how long your home pool is, but I do think that you get a better workout if you are swimming laps that are at least 20 yards long. If you have the typically small home pool, I'm not sure there is enough length to really get a good workout swimming laps. You might be better off with water aerobics and other exercises. The current issue of Cooking Light features a water workout; here is another one from
Self Magazine.