Has he always been like this, or has it gradually got worse?
The fact he does enjoy it when he's there, and that these things are with his friends as well, would suggest it's more about control than anxiety.
DH has crippling, horrific anxiety, and is already stressing about a family wedding next month. But no matter how anxious he is - and I know he is, because he has told me - he has booked time off work, and has worked out ways of getting through the journey and the event in the least stressful way, so that he can still be there with me. He will spend the evening terrified of people talking to him, and focusing on DS so he doesn't have to interact with people.
ExP on the other hand, was a controlling arsehole. He would complain every time I even suggested we do something, whether it was with his friends or mine. I had an important age birthday whilst I was with him, and ended up leaving my own party early, because he was sick of spending time with my friends. He managed to completely isolate me from everyone, in part because he hated social things so much, and I started to feel guilty for wanting to do them.
There is a significant difference in abusive and anxiety based behaviour, and you need to really consider which you think your H is exhibiting.