Good fences make good neighbours children, OP. You need to set boundaries. It’s actually kind to do so. Children can’t cope without boundaries.
Now, I appreciate that’s not always easy and every child is different with regard to where you need to put the ‘fences’, but one thing leapt out at me is that you and your DH are constantly playing with your DS at his demand. It’s good to play with him, but you need to have boundaries and show you’re in charge. Say that yes, you’d love to play X but you’ll be 15 mins as first you need to do Y. Then when you start playing, set his expectations. You’re not there as his servant, you are in charge, so set an end time, either formally or when you’ve had enough (by saying you’ll play for 10 mins more, until the end of the game, etc).
As for your DS ruining trips out, then I think you need to be firmer. Obviously if he’s got a genuine complaint, listen and adapt if reasonable, but basically say that you’re the adults, you set the schedule, and he’s not going to spoil everyone’s day out. If he carries on whinging, don’t tell him off directly, but try making a helpful comment, eg “Oh dear, DH, DS is fussing again. He must be tired and needs an early night” or “We’d better take DS’s iPad/TV away as it seems to be making him behave badly”. You’ll get scowls to start with, but he’ll gradually learn who’s in charge. As he becomes older/better behaved, you can offer him a choice of where to go, but let him choose between two options only, both chosen by you. Don’t let him add a third option. Be very matter of fact about it, “Well, if you won’t choose, we will”.