I used to use skim milk in my shakes but switched to using water about a year and a half ago after hearing a lot of recommendations on this board. I've had an easier time keeping bodyfat in check since then - though I can't be sure it was the primary factor. I still drink about half a liter a day of skim with my morning müsli, love it.
Post workout you want the protein to be easily absorbed and utilised by the muscles.
Milk slows down this absoprtion rate. This is why we take the shake with water.
At all other times you can take it with milk as we want a slower digestion rate at the gut. Casein/Whey is better suited to this.