OP I agree with this: 'I dont think you should think you're responsible for the way he's behaving'.
Also, as you have identified he is being a crap husband as well as a crap father at the moment.
The question is, what do you do about it?
Clearly there are options. One is to leave. Drastic, but it may come to that.
It doesn't sound, yet, whether you have decided if that is the right path for you and your son.
So meantime, you may as well try a few things to bring about the changes you'd like to see. In some ways the two are connected. try some stuff to bring about the change you would like to see in him. If none of that works, you have your answer.
So what might you do to get to the bottom of this?
- have a serious talk. List out what you do vs what he does, do the chart tracking his time spent at home vs yours, all that stuff.
And / or
- just hand over responsibility for you DS regularly over the next few weeks and months, and leave him to work it out.
I hear what you are saying about wanting to spend quality family time together, IMO the route to that is getting him first competent and then enjoying his time with DS on his own. Quality family time will start looking very attractive to him after that.
The period you are in, of it being hard for you to leave DS for longer periods, or wanting to, will pass. But my advice is if you wait until then, the habits between you and your DH of who does what will be more ingrained and thus harder to break. Or, even if he does get more involved, he will never have seen just how hard it is to care for a baby, and appreciate what you are doing and have been doing since DS was born.
Probably you'll do 1 and 2 before you can be in any position to decide if he really is just a crap husband after all and to kick him out.
If I were you I'd reflect on the kind of person he is to decide which way round I approached this. My DH and I have had 'the talk' multiple times in the two years since DD was born. It has been most effective when it has come shortly after him looking after DD solo for a considerable chunk of time (ie a least over a meal or nap time). For others it might be the other way around.
Finally, I know you wish you didn't have to strategise like this. And I know other posters will say you shouldn't have to, that it is making excuses for him etc. All true. However, as someone else said up thread, we live in a world where many boys and men are not socialised at all to consider how their lives will change when they have a baby, or what they should be doing about it. Sounds like you've got one of those. So you either take a fundamentalist view and leave now, or have a serious crack at socialising him first, and then decide whether that is enough of an improvement for you.