I have never had any issues with losing weight while drinking coffee. I stopped adding sugar awhile ago and only a dash of milk. I have read that drinking coffee does have other side effects on your body if you drink high quantities of it. But it doesn't sound like you drink that much so I wouldn't worry.
Hold your head high, don't ever let them define the light in your eyes, love yourself, give them h3ll, you can take on this world, you just stand and be strong, and then fight like a girl.
I don't drink coffee more than a few times a month. For one it tends to give me heartburn. For two just about everyone I work with is obsessed and dependent on coffee and I find it really irritating. I am glad I can wake up naturally, even with very little sleep, and be ready to start my day without the help of caffeine.
Then, when I really need the occasional help, caffeine is much more effective.