The most important thing, in my opinion, is treating your exercise time like an appointment with yourself. If you had a dentist appointment, doctors appointment, a haircut appointment with a stylist that takes 2-3 weeks to get in to see, or something similar-and your brother just popped in wanting a babysitter at the last minute-you would probably say no, you have an appointment, and you can't babysit at this particular time-but you CAN at such and such other time/day instead.
This is what you need to do with your exercise-treat it as an appointment with yourself, and plan other things around it.
Just say "I have plans this evening, but I can babysit tomorrow/Saturday evening/whenever for you, so you can go out."