Well after all these responses I've made my decision!
When I previously tried to go sugar free I was also going vegan and I think I overwhelmed myself with too many changes in my diet. I went back to eating fish and chicken, but I avoid red meat, and I have cheese in various forms but not everyday.
I think phasing out sugar would be a good idea for me. I have horrible issues with compulsive overeating and binging and I almost always binge on sugar and bread. If I let myself have a little it usually doesn't work out.
So here we go again! I don't advocate cutting out entire food groups but I think if something is bad for you and is causing health issues then it's best to avoid it. I do think sugar spurs me onto more sugar and beyond that, even if I can just eat a bit and stop, it's not giving me any real nutrients to work with.
I might as well replace my sugary treat with a fruit-y treat, to actually get some good-for-me stuff.
I have been in a state of mild panic for the last week freaking out about my overeating and binging but I really think I need to try some tweaks to my diet before I declare myself broken and unfixable. There's a lot of evidence showing that some people simply can't tolerate sugar or lots of refined carbs from bread and such.
We'll see how it goes.
Thanks for all the responses!