There's an interesting article on caffeine and blood glucose levels on
WebMD that might be helpful to read. I gather that this is only important if you have diabetes or problems with your blood sugar control.
Some diets have restricted caffeine because of a claim that it can be dehydrating. However, that has been deemed a myth because the amount of fluid you drink more than covers the amount you lose from the diuretic effect.
If you have a problem cutting down, you might try half-caff coffee. Or gradually decrease the amount you drink. If you don't see a problem with your weight loss, then there's no reason to give it up