In the 50s and 60s kids grew up on sweetened cereal, BPJ sandwiches, ice cream snacks, and lots of starch at dinner, and yet North America was far less obese than today. Clearly sugar and starches are not the only culprit (or even the main one, IMO). We eat more and move less. As far as I'm concerned, that's what's making us fat.
I don't eat much sugar on most days, but once in a while I do. Like today, for example: I had a 140-g luxury European chocolate bar for lunch. I know from experience that I won't have any symptoms, sugar crash, compulsion to eat more sugar, etc. If I only do this every few weeks I see no harm in it. (My lab values are all fine.) As I've stated before, I aim for the LEAST RESTRICTIVE strategy for maintaining my weight loss and health. It's what keeps me sane.