Jensassy, I have a Fitbit too and love it. Next month will make 2 years that I've been wearing it.
I wouldn't get rid of it. There's nothing wrong with trying to improve our fitness, and if we have fun with our Fitbits, good for us.
I feel undressed if I don't wear mine every day, and if I don't put it on first thing when I get up, I think of all the steps I'm "wasting".
I don't see anything diet-y about wearing a Fitbit. I'm just a numbers geek who loves to have that kind of data for myself.