I think there is some debate about skim milk, personally, when I drink milk (which I try to avoid cause I get HOOKED on caffe lattes), I drink organic 2% or whole milk.
I try not to go for the any food is bad route, but there are cultures around the world that do not drink milk and seem to do quite well without it.
I would cut back on the milk. Everything in moderation.
I love what Arnie said "Milk is for babies".
My coach has also always told me "Dairy/Milk makes you look soft".