Haha I just praised the glory of yoga on another thread, but oh well I'll do it here too
With the exception of one strictly cardio (bike or elliptical) workout a week, yoga is my only exercise. I do around 4 1 hr 15 min classes a week.
Yoga can range from more gentle stretching/breathing (hatha) to more vigorous cardio (vinyasa/flow/power). Some classes are designed more for relaxation and de-stressing, but a lot of them are not. Fitness-focused yoga include a good deal of strength training: upper body from planks/sun salutations and lower body from standing positions. Most of yoga focuses on the core, so your abs will often be engaged even when you aren't doing explicitly core-building positions.
Yoga has the additional benefit of building flexibility and lung capacity. I see a tremendous difference in my breathing and posture from yoga that I don't see from other sports or activities.
One of the big caveats with yoga is that you HAVE to do the positions correctly. After doing intermediate- advanced yoga for 5+ years, I firmly believe that the majority of the people who say they can't get a good workout from yoga are not doing the asanas (poses) correctly. It can be as small as not keeping your leg firmly straight and bending it a little bit, or not ensuring your thigh is parallel to the ground as it should be. Most of this isn't the fault of the student- you really have to find a good teacher. Good teachers will provide clear instructions on where your arms and legs should be, how deep the position should be, etc. Better ones will go around and correct your posture to ensure you get the maximum benefit. Even "simple" poses like Warrior 1 will get your heart rate going and your thighs burning if you hold it the right way.
One of the best teachers I ever had only had us do 10 basic poses over the course of the 1.5 hour class. Because we had to hold them for a long time and in the exact right position, pushing our bodies further than they wanted to go, they were exhausting. My heart rate would be sustained at 70% for most of the class and every muscle score the next day.
I have always run/been a competitive athlete in very good shape and I find I get just as good a workout with yoga as with other sports or exercises. Try a class and if you like the idea and feeling of it, stick with it. Try different teachers too. Admittedly the ones who work at gyms are not the best, but yoga studios are expensive so sometimes they are the only ones accessible. There are some good ones out there though, so hopefully the one at the Y is good.
Happy yoga-ing