I wouldn't jump to any decision.
Things are away to change anyway as your three year old will soon be doing longer days.
But really I'd be looking at what you are doing, what is absolutely necessary and what can be done at different times.
I don't spend my day off cleaning. I'll do it until 10 then I am going for a walk, swim, run, meeting a friend etc.
I can run a cloth round the bathroom while a child is in the bath, hoover once they are home from school, do a quick mop after tea.
What I can't get back so easily is that time for myself.
Allocate the office work to an evening and tell your husband he is doing bed time.
Can you ringfence family time golf mustn't impinge upon? Are there times where him playing a round makes less difference because you are taking kids to activities, have something you want to watch on telly, have a friend popping round?
I think you all just need a reset in terms of how you do things. This clearly applies to him but I do wonder if it applies to you too? Your three year old isn't a baby and doesn't need the constant input of a baby. Do you need to rethink how you spend your free time too?