Hi, Caseyboo!
I also exercise 5-6x a week (in my gym) for an hour at a time. Then I take classes (yoga and Tai Chi) 3x a week. Then I go rollerblading with my husband during the week, but that's just for fun and I don't consider it exercise (it is, just not formal). It sometimes works out that we blade on the days I'm not in the gym, but since it's just fun in my mind, I don't worry about it.
I'm also a personal trainer, certified by ACE (American Council on Exercise). The "have one day of rest" thing is entirely right. If you are doing a formal, challenging workout, then your body needs that one day (or two days) to recover. When you're strength training you're tearing muscle fibers microscopically--they heal themselves a little bit stronger, and that's how you develop your muscle strength.
Probably what your husband is getting at is the idea that our bodies are made to move, not to sit in chairs all day. If what you do on your 6th and 7th days is regarded as "fun", if it's just playtime for you, if it's not formal and rigid, and if it doesn't involve any intense muscle development, you're fine to do something. No one's saying you have to lay in bed on that off day! Take a walk if you'd like, go to the park and play, ride your bike around the neighborhood to visit a friend. All that is okay. But definitely take a break from muscle work (weight lifting). And, if your body is telling you "I'm exhausted, please don't make me play today. Let's take a nap," then listen to it and respect what it's telling you!