Coffee only has 10 calories per 20 oz. Unless you're having, as mentioned above, some kind of insulin resistance issue with it, it's highly unlikely that the coffee itself is making you gain.
However, if caffeine makes you jittery or upsets your tummy and you wind up eating something because of that, then it's not really the caffeine that is derailing your plan so much.
We're all different however, so you have to find what works for you. Whether it's physical or mental ... if you think it's derailing you, then try cutting back on it and see what happens.
.