Yeah, it's normal.
I still feel a bit guilty even on planned rest days. I just really enjoy exercise.
I made it work by setting a baseline of must-do. This means my normal routine is to work out M-F, specifically to spin M/W/F and lift T/Th. I can do extra on top of that, but if I'm ever too tired or sick or just whatever, I fall back on the basic routine. Then when I take a break, I cut myself some slack.
So lately I've been doing a Saturday morning run, but it's above and beyond my basic routine. Yesterday I skipped it. It was just too cold. Guilt set in, but I was able to nip in the bud reminding myself that a run yesterday would have been extra.
That's how I deal with guilt, anyway.