It seems like you are the default parent. It's interesting that you say he does two school drop offs a week, so that you can go the gym. Shouldn't he be doing half the drops off anyway? So it's hardly doing you a favour, if he's not even doing his half of the childcare/chores.
I agree that having a half day a week to do his hobby is not unreasonable. But the fact that he is making decisions to just go away for the weekend without discussing it, when it sounds like he is already taking you for granted, is totally unacceptable, and I can see why you are getting to a point where you are becoming resentful of even the mention of golf. He definitely knows he's wrong on this, or he would have had no problem discussing it with you beforehand.
My DH went through a similar phase - playing cricket every Saturday and Sunday, and sometimes in the week as well. Not particularly expensive, but very time consuming. Plus football in the winter. Kids were older than yours, but it got to a point where we never saw him, couldn't make any weekend plans, everything at home fell to me. And then he had the nerve to moan about me going away for a weekend with a group that I volunteer with. After a big row, we sat down and I explained to him that he was taking me for granted, I seemed to have become the default parent, how resentful I was becoming, and how damaging to our relationship the whole thing had become. We resolved it - he now plays once per week, and only if we have no other plans. It's much better.
OP, I would sit down with your DH and have a chat about how you resolve this going forward - how much time it's fair to spend on his hobby, how it should be financed, how you are going to split childcare etc from now on. It's not fair that it all should all fall to you just because he wants to spend so much time playing golf.