Exercise is not a waste of your time unless you are trying to do it too hard, too long, too much, and too often.
Try not to think all-or-nothing. There's no reason to go from zero to 60 in 10 seconds!

Shows like Biggest Loser make it look like you have to kill yourself to have a "good workout," but it's not true.
Set some moderate exercise goals for yourself. It's nice that your BF wants to do it too, but you may find that his workout level is higher than yours. You might want to exercise on your own occasionally.
And no, you won't see immediate results--so don't think you're "failing" after 4 days.
Good luck! Keep moving!
Jay