I don't track sugar, but I do track carbs. I think it's better to eat the whole fruit, but if you have trouble with blood sugar, you may still want to minimize fruits. Or keep it to lower sugar fruits, such as berries. If I eat a fruit smoothie, I add protein to it. I also don't drink them very often or drink a huge amount.
I think it depends on what plan you're on to lose weight. If your weight is stalling, you might want to try cutting down the smoothies and see if it helps.
Also, do you have diabetes? That might be a different issue. In that case, you may want to consult with your doctor to see how much is too much for you.