I would aim to eat a protein/carb item before you workout, the banana may not be the best option. After you workout, you can also have a snack.
You aren't nullifying your workout because these are calories that fit into your day, not calories you are adding extra. Also your workout is more than burning calories, it is building muscle, improving cardio condition and burning fat.
You can't out-exercise poor eating habits.