No, it wasn't an accident that he joined just before the baby arrived.
It's called "forward planning" and is typical behaviour from selfish men who want to avoid new babies/family life/sharing responsibility for the life he created.
You have most likely had your baby with a dud/waster/twat.
This is who he is.
Unlikely to pull his weight and step up, because he is selfish and he will always be inclined to put himself first.
My advice is be very careful.
Don't rush to have a second baby with him.
Keep an account with money in it that he doesn't know about.
Make sure that you maintain your career and even look at going back to work FT.
Men that have the forethought to join a sport like golf just before a baby is born are the very type that will play away.
I'm sorry if that is harsh.
Disregard what I have written if you like, but protecting yourself and being wary is a wise move here.
Bottom line is he wants to limit as much as possible the amount of time he spends with you both.
I'm sorry and hope to be proved wrong.
You definitely need to start taking a lot more time to yourself asap.