Oh PLEASE leave him.
He is crap. He is nasty, childish, disloyal, arrogant - he's not a keeper, he's not a good partner, he won't be a good dad.
You say you've had issues and you thought that the pregnancy had made him grow up a bit - but now you can see that surprise surprise, people don't change!!!
The house is yours. If you stay married, before too long the house will end up being half his too... sorry to say it but that could be a major factor in why he's been keen to marry and have a baby.
It may be hard on your health in the short term to split but imagine a LIFETIME of this. Of constantly feeling that your supposed life partner is slagging you off and laughing about you behind your back, talking you down to whichever 'friend' is the flavour of the month. That kind of shit REALLY shortens your life, believe me.
Wait until you're stuck with a newborn and your supposed husband is busy in a bar with his friends saying he'll be home soon but you know he's pulling faces at the phone and rolling his eyes and his friends are sat there sniggering at you 'She's been a nightmare since we had the baby, probably still feeling the stitches I guess, haha!'
Can you imagine? 
GET RID.
Please do so quickly so you can a. decide whether you really want a baby with this guy and b. whether you want to stay married long enough to effectively hand him half the value of your house when you finally do get too sick of taking his shit.
Oh, and you're the bad guy to his family? Um, so what? How about you tell his sister that no, he hasn't had an affair, he just happens to be a nasty, disloyal, unsupportive sack of shit and you're beginning to realise it, which is plenty bad enough for you to be thinking right now that he isn't worth it and you could do better.