Years ago, I remember reading about a woman who was 100+, she started exercising by walking to the end of her driveway and back. Everyday she would go a little further. Her main problem is she never exercised before that and so just a little exercise helped her.
I was just thinking about this tonight, but everyone has their own fitness level and it is dependant on how much fat your body has, how much exercise you've done in the past, what medical issues you have and your age. So if you feel like your exercise is challenging you, then that is all you need to do. Slowly increase and you will build up your endurance.
A couple years ago, I first joined a gym and discovered ellipticals. I started at something like 5 minutes and would slowly increase my time. There came a point when the incredible happened, I realized I could go on forever. I was doing 35 minutes of elliptical almost a year later. After gaining a few lbs and not exercising for a little while, i'm back to building up my endurance. I am now at 14 minutes on the elliptical.
No one expects you to run a marathon, but you could build up with that and it could all start with something like walk off the pounds