I'm another one on the "you don't have to give up coffee" team. You do need to limit your sweeteners to a limited number of calories, however. 1 tsp of sugar, however, only has 15 calories. If you had two cups a day, each with a tsp of sugar, you'd only be adding 30 calories to your day. No, those 30 calories aren't the most nutrient-laden ones, and there are better 30-cal choices, but it isn't that many total, so it really might not be a problem for you.
Have you experimented with different sweetners and/or sugar free flavored syrups? Monin makes a ton of sugar free syrups in all manner of flavors, if your opposition to artificial sweeteners has to do with taste.
There are actually a lot of studies which have shown health benefits to coffee consumption...check out this summary from WebMD.
http://www.webmd.com/content/article/80/96454.htm. A few of the benefits: a reduction in diabetes risk, a decrease in risk for Parkinsons, gallstones, colon cancers, and others. There is no reason that coffee can't be included in a healthy diet unless you are sensitive to the caffeine or it stalls your weight loss.