The connections between dietary fat and heart disease are more tenuous than scientists once thought they were. Remember in the 80s when all dietary fat was treated as toxic waste? Research on low-carbohydrate eating (and there's a lot more of that than there used to be) suggests that fats--including animal fats such as those found in whole milk and butter--are not the "heart attack on a plate" they were once presented to be. Then there's the whole "French paradox" thing: traditional French cooking is loaded with animal fats in butter, cream sauces, pork fat, sausage, full-fat milk in cafe au lait--yet the French rate of heart disease is demonstrably lower than that in the U.S.
And then there's the Maasai, whose diet is largely made up of meat, milk, blood and grain, yet who have half the blood cholesterol levels of the average American and have virtually no history of heart disease.
With that said, if you're watching your calorie intake, you'll have to choose between larger portions of lower-fat milk or small portions of full-fat milk. It's up to you to decide what you think is worth it. Skim milk tastes like water with a little chalk stirred in to me, so I empathize with your love of full-fat milk--but it does pack a caloric wallop. It's the only thing for proper cafe au lait with chicory, though.