I've been reading and posting on this thread and I'm not surprised you're upset Flossy.
I've witnessed a couple of non-resident fathers behaving just like your ex husband. They make a new life, and expect their children to fulfill the role they want them to play in their new reality. The fact that for the majority of the time the child lives in another place and has a whole other life is irrelevant to them. They don't want to have to engage in any of it. Mostly because for them it's boring and they're not interested in knowing the child's friends, other parents at the activities, or being part of that other life.
Yes, lots of nrp's aren't like this, but lots are, and the OP's definitely is. He's moved further away and I bet if it suited him he'd see his son less. Frankly f**k him, let him take you back to court if he sees fit. I suspect he won't. That would take effort and commitment on his part. If he did do it, from what you say it would probably only be to try and get at you.
Your son's contact time at his dad's sounds boring and miserable. Much better that his dad takes him to events that are an intrinsic part of HIS life. The fact he isn't even prepared to consider doing so, when he has nothing else arranged that would make it impossible, shows what an arse he is.
I don't think this is controlling on the part of the OP, or micromanaging the other parent's contact time. It's just common sense, fair, reasonable, and what you do when you love and care about your children. No wonder the OP is stressed out. Her ex husband has been violent towards their child and can't separate his feelings towards her from those towards his son. Thank god the child has such a great mum in the OP.