I think regardless of everyone else's experiences, you have to gauge how your body reacts to it. When I'm following core, I allow myself FF/SF Yogurt, as well as low fat mayo instead of FF. Neither of those foods are trigger foods for me so I don't abuse them, and still have success. But as I said, it's all about how you use the food and how your body takes to it. I think the first week, in my own experiences anyway, Core is such a change for the body that the LF milk shouldn't be a problem. However you may see it effect you, if not the first week then later on down the road.
Sarah