As NYC said: try to eliminate it and see what you get... Milk, skim or otherwise, is inherently pretty high in sugars, and i avoid most dairy for that reason: i would rather get my carbs from things that have texture/crunch/greater mouth fun.
That having been said: does milk fit into your daily macro projections? if it does, enjoy it if you want it, if it doesnt, kick it the hell down the line.