I always treat my coffee like a treat. I figure it's my tiny splurge item and if I make room for it in my calorie count and keep my fats low in other places then it's ok. Besides, according to studies, coffee drinkers are less likely to have type 2 diabetes, parkinson's, dementia, few cases of certain cancers, heart rhythm problems and strokes.
I'd go ahead and treat myself to a cup of coffee or two. Maybe, cutting one out and replacing it with a cup of tea or a glass of water would be a step in the right direction?