Of course you all realize that "dairy" also applies to whey protein and casein/cottage cheese. I hope you're not going to cut all of those out of your diet as well, though real food is always better than pulverized and instantized stuff.
If you're not allergic/lactose intolerant then lactose is just another carbohydrate, and low GI at that. However, if you have problems with lactose then don't eat it! There is some evidence that components of dairy foods are actually insulogenic and result in a higher insulin response with a mixed meal (this means the GI appears low because your pancreas produces extra insulin). This may be a good or bad thing depending on whether you're a BB with good insulin sensitivity or an overweight insulin resistant couch spud.................