I do it at varying intensity 4-5 days a week, but I think 3-4 times a week is enough, too. I do a mix of yoga classes and videos at home.
My exercise schedule depends on my wacky rotating work schedule. I aim to run 2-3 times a week, go to yoga class 2-3 times a week, and do a yoga video at home the remaining days. For example, last week was 2 yoga classes, 2 runs, 3 yoga videos. This week so far has been 2 yoga classes, 1 run, 1 yoga video, and 1 missed day, planning on yoga class tomorrow and run Saturday. But I'm trying to lose weight, increase my fitness, and destress, which is why I'm trying to exercise every day.
For your goals, aiming for 4 days a week and being happy with 3 days a week sounds pretty reasonable to me. But I'm no expert, just a recovering fat chick.