I think we all respond differently to certain foods. Milk has an interesting combo of proteins, sugars and if its not skim milk, fats. Skim is probably hardest on blood sugar, as there is no fat to slow down the absorption of the sugars.
As we always say, keep testing. If you eat a food, test afterward, and see what your response is. Many people do well with whole grains, but I can't eat wheat especially, even whole and sprouted grains. Many people find increasing dairy helps them lose weight, so check and see what your response is. Since you seem to enjoy it, I hope its a good food for you.
zen and the art of weight loss, finding the true path of en-lighten-ment