I'm not sure what kind of plan you're looking into, but if you're going with calorie counting, then I would suggest to either:
1. Switch to skim milk, almond milk (yum!), or soy milk, OR
2. Lighten up on (or eliminate) how much you use, OR
3. Leave it the same and just count the calories. Half a cup of 2% milk is about 70 calories according to my calorie tracker, and that's really not bad at all.
You mentioned the fat content, but 2.5 grams of fat isn't make or break.
Now, I'm assuming that you mean that you're only doing this once a day. If you have 5 of these a day, that might be an issue!