OP, you are in a tricky position here and it’s all very well people just saying, “F him.., just go out at 7am etc etc.” It’s not that easy and these things are insidious.
The fact is, he won’t stop playing the golf because, in his mind, he feels 100% entitled to this and nothing will shift that mindset. It sounds as if he has little experience of parenting the kids in his own for any length of time?
A lot of men are crap at staying home with children because they find it too claustrophobic and they are not in the mindset for this. My DH was like this, but Fortunately, I found he was much better if he could go and “do something” with them - ie, take them out on bikes, or to a museum or camping whatever. There had to be a point and a purpose, basically. It was the mooching or unplanned times inbetween the activities he couldn’t cope with (that fell to me and why I never really felt I could plan anything for myself).
Your husband may be similar that he finds it difficult to do the humdrum part of parenting. But he does need to find his own way of being with his kids - otherwise his relationship with them will suffer long-term. Does he ever take them out for instance (without you)? Is there an activity that could be “his thing” to take them to on a Sunday - eg. swimming or whatever?
A lot of men find younger children difficult to entertain and this is why you sense they can’t be left with them. You don’t feel confident enough to leave them and you feel that it will put your husband in such a bad mood that you don’t push it. I know this feeling. They don’t even have to tell you you can’t go out, you just know you can’t because you are conditioned into thinking you have to always be there as default. I was like this for many years and its a conditioning that’s hard to shake. You felt guilty leaning them in a situation that may anger them do you take it on to not make waves.
Go out on Sunday. Has he now agreed you can go 2-5? Well go and he will probably manage fine. Then tell him you are a human being and you, like everyone, need some hands free time away from the home and family. Tell him you feel energised as a result of those few hours out and you will be a better wife and mother going forwards if he will support you in having some time. I hope he can’t start to shift his mindset. Good luck!