Personally, I love wheat bread so I wouldn't consider it suffering to eat it. White bread is basically void of nutrition. Whole grain bread (100% wheat and others) on the other hand is good for you. It depends on what your focus is. If your focus is purely calories, then eat what you want based on the calories you want to achieve.
You can't out-exercise poor eating habits.