Myself & DD are both lactose (milk sugar) intolerant, due to related health issues she sometimes is dairy intolerant, as in WHEN she is having a flare up of her stomach/intestine problems,
so far diagnosis her is IBS, but as she wasn't having a flare up when we saw the specialist, so apparently they can't be 100% sure, as during a bad antibiotic induced flare up, we ended up down A&E with suspected appendicitis, but Dr there said colitis, so its wait & see & have been told I'm managing it well with her diet etc.
in short, from my experience with both myself & DD, during times when you are not experiencing problems, you will get away with eating some diary (in our case wheat too) but if you eat a lot, or have other factors such as stress, antibiotics etc mess up your digestive processes, then you will have problems with eating any diary.
In our case, neither of us can drink milk at all, & do get the bloating you describe if we do, we can drink "Lactolite" lactose free milk, without any problems....UNLESS digestion is already disturbed as mentioned above, then ANY diary will make it worse
hope that makes sense
if Lactolite isn't the answer for you, then theres plenty of other good alternatives, Almond milk being the best, as like cows milk its a good source of calcium, but you can get that from other sources such as fish etc....DD loves coconut milk for shakes etc
also your gut flora will be out of balance, so you can buy powders such as Viridian gut flora powders to add to smoothies, shakes etc, these will help your stomach/intestines settle again
as an adult, I swear by cats claw, but its no good for kids