OP,
"He brought him swimming for YOU".....there is your problem.
That you would think like that.
Surely he brought HIS son swimming for HIS son, not you.
He sounds selfish and lazy.
He sounds as if he is doing very little in sharing the load.
Why isn't he doing the weekends.
You have very low standards, that you would accept such selfishness.
He sounds like a shit partner and a shit father.
He should be doing far more and he definitely should be giving you extra rest as you are doing the night wakings and working.
Don't have another child with this selfish man, he will only get worse.
You have made a very hard bed for yourself by tolerating this bullshit.
Watch your health, women with shit partners often get sick through doing it all.
Oh and not good that you are nervous of mentioning it.
Not good at all.
He knows bloody well that he isn't doing his bit, but he doesn't care.
A good man would do more.