I'm not going to base my answer on any research or anything like that, so take it for what it's worth.
I think you should go based on how your body feels physically. If you have energy and feel good, you shouldn't take a break just because you think you should. Same goes for rest. If you are truly worn out, your body needs some time to recover. I think our bodies are great at letting us know what we need most of the time.
As far as feeling like it's not worth it....I get it. Maybe you could use a pedometer to track your steps? My husband has been using one (actually, it's a fitbit) and it really motivates him and makes him feel like he has accomplished a lot since he likes numbers and it gives him something concrete he can review. Just a thought.