I can't see anything to indicate a "midlife crisis" in all honesty. He just sounds like a parent who is relishing the freedom that having older children brings, and pretty much like the friends (both male and female) I know with similar aged children or older. I wonder if you're looking for reasons to leave him and have latched on to this? Are you happy in the relationship aside from this?
Of course it's still important to set aside time to be together as a family, so there's no harm in setting aside certain meals or times of the week where you all do something together. However you now have the time to not only focus on spending time together as a couple, but also rediscovering who you are, and doing things that you want to do, either on your own or with friends.
Why didn't you go with your DH to the neighbour's? Circumstances such as SN or ill health aside, your DS should be fine on his own for a while, especially as you were so close. Even if you hadn't planned on going, you may have enjoyed it for an hour or so?
I also wonder why you don't go out with your DH occasionally when he has something planned. it may not be your cup of tea, but worth trying once or twice? It sounds like he wants you to go with him. Of course there are always things that need doing in the house, but to be honest those are almost neverending, so could the four of you get everything done on a Sunday morning, so it frees up time for you all?
It sounds like it's also time for you to have the enjoyment of a bit of time to yourself, doing something you enjoy. Do you work long hours in the week, or could you spend an evening or two doing something fun, whatever that is for you? Do you have an interest that you would like to pursue but haven't had the time for before?
I am really concerned that you can't listen to him talking about this because you actually want to cause him harm. Of course you may be using this as a figure of speech to emphasise how you are feeling, but it doesn't sound healthy to me. I think the two of you do need to sit down and talk to each other about this, but also more importantly, listen to each other.