Declining the treats doesn't seem like the biggest problem you have here... and I'm ignoring the whole issue of your goal weight, since that's been covered quite a bit already.
There will always be people who say that you're too skinny or too fat or weird or on the wrong diet. People who make fun of your food choices. Some of them will mean it, and some of them will say it out of jealousy or to be manipulative or some other reason. Either way, that's about THEM, not about you. It's their problem. You can't take their comments to heart so much -- you shouldn't be left wanting to eat more or not eat at all just because someone else is being a jerk to you. I think you need to work on your confidence and learn to stick to your (sensible and well-thought-out) plans even when other people don't agree with them.
It sounds like your colleagues might be jealous of your self-control, or that they feel guilty eating the treats themselves and want as many other people to join in as possible. BUT I don't think it should matter why they're doing it -- you're eating healthy for YOURSELF and need to focus on that.