An affair is a definite possibility. However I would say there are two people here looking at the same situation with different lenses.
From his pov YOU may appear the selfish uncaring one.
From his pov it may go something like this: He does a job that seems to be stressing him. You moved into a bigger house, which mortgage he is totally responsible for and you do not contribute towards, with bigger bills that he covers and again you do not pay (yes I know you do the shopping, but that is not quite the same pressure as a debt although it is a pressure to be sure). He has a second child coming with more expenses and since you had the first baby, you have consistently said you are unhappy shouldering the burden and are not happy with his performance as a father (in terms of caring and spending time) And now you are having a second one, which means (to him) more of you complaining about the pressures and not being happy. He feels inadequate to be a father in the way you want (and to be fair most people would expect) and probably unconfident about fathering a second. He goes to the gym to unwind and get out the stress before he comes home and unleashes it on you. But you don't understand, and you think it's just to avoid you. The stress is now so bad he feels the need to smoke to unwind. You complain to him about the smoking habit and the gym, but not about what is making him feel the need to smoke and use the gym. You complain if he goes to the pub (child in tow) to unwind but then say he should support you, so you can have a day to unwind. He feels you're not hearing HIM.
Now, you can say it's unfair and his viewpoint (assuming I'm in any way correct) is wrong and he's selfish, but if any of what I said is true, you both have terrible communication problems. You are talking totally at cross purposes and neither of you feel validated. Men typically switch off and close down when they're unhappy and let It build up until they come out with the 'I'm.leaving you' It isn't right it's really lousy. The switching off and the distance puts us into flight or fight mode and we get angry and hurt.
They say counselling never helps. But I think you definitely need emergency intervention on the communication front. If you cannot find a way to communicate then this relationship is not worth saving. Cut your losses now.
Think also: Are you able to make any compromises? Do you need the bigger house? You said that he throws back at you that he bought the house to 'make you happy' so perhaps he's confused about which one is your priority? The being present at home or the lifestyle? This can be worked out with communication.
I've assumed some things regarding finances so feel free to set me straight.
I will say you do not sound like you particularly want to continue in the relationship and you both do not seem to have the values in common. When values aren't aligned in a relationship it is very hard so think very carefully if you really want to try and salvage it or exit now.