I have never had any issues with losing weight while drinking coffee. I stopped adding sugar awhile ago and only a dash of milk. I have read that drinking coffee does have other side effects on your body if you drink high quantities of it. But it doesn't sound like you drink that much so I wouldn't worry.
I don't drink coffee more than a few times a month. For one it tends to give me heartburn. For two just about everyone I work with is obsessed and dependent on coffee and I find it really irritating. I am glad I can wake up naturally, even with very little sleep, and be ready to start my day without the help of caffeine.
Then, when I really need the occasional help, caffeine is much more effective.