Sorry to hear that too, I know it can be frustrating to not feel safe like that. Would you try some self defence classes?
Well in that case would you be able to do a workout inside your own home in a plan B situation? Sometimes I get a Zumba video on youtube and do that for a while and mix in some different exercises and do lots of squats, lunges and some yoga too. Or you could invest in some home workout system. The only piece of equipment I have is a little stepper with bungee chords but I actually really like it and it was cheap (60). I usually just use it on days when the weather is rubbish and can't go outside.
It's lovely that you guys do your exercise together but I think it's good to have a back up plan for yourself too just in case