I also swam competively growing up, here is my take.
Swimming fast and longer distances is hard work. Not everyone learned how to do proper strokes and not everyone is capable of going fast or going long. Since most aren't, it probably isn't the most effective calorie burning excercise for most.
When I competed, we did around 3500 - 4000 yards in practice 5 times a week - this was just as a kid/teen. Actual people who compete in college or olympic level probably average 10,000 a day. There is NO WAY any of us could get anywhere near that level with work, school, family, etc. It's a full time job!
So I guess my point is, swimming is great exercise to increase aerobic ability, lung capacity, flexibility and strength. And if you have an hour or two a day, and you can go at a fairly good pace, I am sure you will get a great calorie burn. But if you don't have an hour or so or you go super slow without the best form, you might want to do something like running to burn the most calories.
I still wouldn't give up on swimming - it's low impact, you can do it until you are 100 years old and it is great for flexibity, lung capacity and muscle toning. It's just that there are other excercised that you could do for less time and burn more calories.