A couple of things here;
Your DH is gaslighting you. He is doing what he likes, whilst you shoulder the full weight of your 2 DC’s needs and then he is turning it on you, when you express concern over it. He has you in a nice little corner doubting yourself and justifying it all in his mind.
It might not be a full on affair, yet, but it is an emotional one. I bet his little walks are all about him, and how hard it for for him back home whilst Vera, his side kick, is listening and making him feel better.
Walk on a golf course, with no one around means privacy and no prying eyes. My DS and his GF are always “going for a country walk” from our house and I bet my life it is a for snog and a feel (and more) with no one to account to.
As an interim solution to ending your marriage, which I feel you are not at yet, I would be sitting your DH down now, and telling him the following;
“I know that you are meeting with Vera on Monday at the golf club for a walk. You say she is a friend but the secrecy and lack of introduction makes me think otherwise. You know how I feel about this, and I’ll leave it to you and your conscience to decide what kind of friendship this actually is.
However, what this has made me realise, is that I am very unhappy and stressed out. Something has to change. Like you, I too need friends and people to talk to, and to offload to. What I want now, is to go back to work, at least p/t and have more free time, like you. We need to have a serious discussion about this, and how things are going to change around here”.
Whether or not your DH is playing fast and loose with your family life this is a wake up call here. You need to get YOURSELF in a better and more secure place, for you, and your DC.
I found Mumsnet just a short time after it started. It has taken me through many ups and downs. The women on here know what they are talking about. One of the best pieces of advice I have seen is “If you can’t save your marriage, save yourself”.
My DH dicked me around a few times when I was a SAHM. The last time was 4.5 years ago. I immediately went and got a p/t job (that I can convert to FT anytime I want), got a pension, set up a savings scheme, took up new hobbies, and invested more in friendships and a network. I am still married, but I’ll never let myself be in a vulnerable position again.