Be glad you haven't been foolish enough to marry this arsehole.
You are being called selfish because he wants to shut you down.
You got YOUR baby and now you need to STFU and be skivvy aupair to HIS children.
You are clearly very naive and cannot see that you have been sucked into a skivvy role.
He's not happy because the staff, YOU, are kicking up.
Your baby is two months and he has set out his stall.
He continues to live his life and has his golf day every week, whilst you do the skivvy parenting role that he has zero interest in.
12 months from now you will see this selfish waster a lot more clearly than you do now.
Be so glad you never married him.
If you were MY precious daughter I would be telling you to return to work full-time and be looking carefully at YOUR future.
That is if you don't want to be parent to 4 children with this waster, and enjoy your baby years with YOUR child.
Ask yourself would you have had a baby with him if you knew you were going to be parenting HIS 3 children full-time.....while he avoids family time like the plague???
You have options, look at them.
Can you go and stay with your family for a bit and let the loser crack on with his children?
Think carefully of the type of man that calls a new mother selfish because she wants a few hours to herself.
Its ok for HIM not to want to be with HIS children, but you have to suck it up and have zero free time?
He really does believe you are a mug with zero self respect, that will accept being treated like a skivvy aupair.
Wake up and protect yourself and your new baby from this selfish loser.
This relationship is doomed.