Has he been off sex since the baby arrived? Perhaps he doesn't want another baby?
Perhaps he feels overwhelmed by responsibility? Or is silently blaming you for giving him less attention or feels he doesn't want to be a family man but feels he can't tell you and is just being resentful instead? Either way he's not happy, and is being passive aggressive about it.
Could be something physical, but if that were truly the case he would be feeling as frustrated as you, and keenly motivated to sort it out and pursue options.
People always talk about physical and emotional abuse, but unilaterally withdrawing sex without disscussion can be every bit a form of abuse. It's very controlling, cruel behaviour. The partner watches the other partner suffer. Watches while they jump through hoops, trying to please them, cooking their favourite meal, putting extra thought into things, arranging dinners, putting extra effort into their appearance, going on a diet, losing weight, buying and wearing sexy lingerie, attempting to initiate sex multitude times. Then brushes them off and rebuffs them. They're not stupid. They know what they're doing and how shit it's making you feel. You get into bed naked - and they just turn over without even explaining why they have a problem, leaving you feeling humiliated and ashamed. Because they feel they have control in outright rejecting you. It's cruel, selfish, controlling behaviour.
It also puts the entire weight of responsibility for the relationship on you. You are the one left feeling trapped, the one having to question whether to break up the family as they will not even acknowledge what they are doing. You ultimately are made to feel like the guilty bad guy for questioning if you should stay or go for something seemingly trivial. It's not trivial, about losing a deep form of human connection and physical validation and acceptance.
Unless you openly choose differently, sex is very important in a relationship. Even in cases where some sad unforseen event occurs intimacy remains important. But then such situations don't contain the element of cruel rejection you're experiencing.
I say confront him and if you get nowhere leave. It won't improve because he'll be showing you that the respect for you is gone.