Tele is right about it needing to get straight to an ice bath. Ice baths cool the milk much more rapidly than the refridgerator.
Milk is a warm protein, which means it will start growing bacteria VERY fast, and every precaution must be taken to prevent it. Right after I'm done milking my cow, I will rush to the house to get the milk chilling even before I release my cow from the stanchion (she stays content because she's still finishing her grain:) )
Often times that bacteria is what creates a foul, bitter taste in the milk, and the products made from it.
About the freezing, I am not sure. My first reaction to that was, "surely that can't taste good." But I've no evidence to back that up, so I am of little help there. My mom is very knowledgeable about fresh milk, so I will ask her and get back to you. :)