Milk has developed over 310,000,000 years. It's the only thing besides eggs that is a full-spectrum food that provides every nutrient that a growing body needs. We now kill everything in it, the good and the bad, in order to protect from pathogens that can cause illness.
Illness is bad. But....
The primary indicator that milk has been pasteurised is the inactivation of alkaline phosphatase. This has recently been found to be a primary preventative factor against asthma and alergic diseases. Asthma kills over a thousand people a year in the UK. The allergic protection from raw milk doesn't just extend to lactose intolerance, but a large range of other allergies (confirmed by skin-prick test). Enteric Coated Lactoferrin supplements have been found to reduce median obesity. The fat globules in milk provide enteric properties (they get the good stuff to the gut and it's not destroyed by stomach acids). Lactoferrin, in a study in human breast milk, was 35-90% inactivated by pasteurisation. Both Lactoferrin and milk oligosaccharides have been found to improve intestinal barrier function, so while e-coli O157:H7 (which produces Shiga toxin which can lead to organ failure and death) is not killed off in raw milk as it would be in pasteurised*, it's also less likely to cause serious health problems for raw milk drinkers as they benefit from protective effects.
Raw milk also has antiinflamatory properties. This provides significant preventative effect against infection and fever.
The Advisory Committee for the Microbiological Safety of Food rank the risk from raw milk to the public lower than pasteurised milk, as outbreaks from raw milk are small (due to the low distribution) whereas 2 failures in pasteurisation in the US, for example, affected 400,000 people due to the mass distribution of pasteurised milk. It's far more likely that pathogens introduced to a bottle of milk will incubate far more successfully in the pasteurised milk than in raw milk, which has tons of microflora in competition with it (so don't drink from the bottle unless you're going to finish it).
The risk of illness via pathogens in milk are approximately 150 times higher in raw milk than pasteurised. That's a lot - but the risk is miniscule in pasteurised, and it's 150x a miniscule risk.
The biggest risk from raw milk that's taken from an established producer of raw milk (ie, not from a tank of milk that's destined for pasteurisation) is from allowing the temperature to rise. Milk is evolved to be digested within 45 minutes of it leaving the breast. Raw milk producers will filter and chill this as soon as possible after it leaves the udder. However, transporting it in warm conditions and leaving it for ages before drinking appears to be the biggest risk factor.
The composition of cow's milk is very similar to human milk (probably the same for goats etc, but all of the information above is from published articles, and I don't want to deviate from it). The CDC strongly promote breastfeeding, and give advice on breastfeeding while ill which largely comes down to "don't breastfeed if you have Ebola or active Herpes Simplex Virus sores on the breast" **, yet they strongly advise against drinking raw milk due to the risk of pathogens. That seems hypocritical to me - the milk's good for you or it's not.
- Raw milk doesn't contain e-coli O157:H7 without contamination. A report on an outbreak caused by raw goats-milk cheese confirmed that the pathogen existed in multiple places around the farm, in the faeces of the goat that produced the cheese, but not its milk.
** TB isn't mentioned as regards breastfeeding by the CDC, but the advice I found elsewhere about human mothers with TB is to breastfeed while ensuring that you're not breathing on your child as it's an airborne virus, which implies to me that you don't get TB via milk, but from contamination of it - which would be killed off by pasteurisation, but also which wouldn't be there if it's from a good herd and taken according to good animal hygiene and milk handling practices... which are things that raw milk producers will need to ensure given the amount of liability they take.