He has to follow the same routine, I I dared fed and bathe them without him he'd make my life living hell.
I'm not allowed to so anything
He has a diary we keep to count down to any up and coming things, even as little as shoe shopping.
To be honest some of this sounds very Aspie / PDA -ish to me. Controlling everyone’s routine not just his own. Needing so much advanced warning and structure. And the count-down diary is a very sensible strategy - you’re doing the right things.
Children’s mental health professionals in some countries are more aware/accepting of high-functioning autism, Aspergers and PDA than in others. PDA is not all that well recognised even in the UK but given his other behaviours even in areas that don’t recognise PDA he might probably be recognised as a variant of high-functioning autism or Asperger’s and you (and he) might get suitable help.
I second the advice to look at Explosive Child meanwhile - it may help.
As for people making remarks about the OP’s DH, trying to raise children with undiagnosed SN and challenging behaviour can put a huge strain on relationships. Our child-psych made a point of checking how DH and I were getting along and that we were getting some rest time together without DS because we needed to be stable together to support DS.
And OP, defo head over to Mumsnet SN children for more support. There are no easy answers and I can’t offer advice myself because my DS is an only child but he would certainly have been hell on wheels around younger sibs. And oh-oh, I still hate shoe shopping with DS. We can usually get by if we go at a quiet time and take the very first pair that fits and go for tea and cake afterwards! Not everyone is that lucky though. For what’s worth my DS is now doing very well as a young adult heading for university, so there is hope 
As for whether he’d do better at his grandparents, if they can give him uninterrupted routine and it’s all nice and regular and predictable, and it sounds as if he is close to him anyway, that may actually work out. I don’t know. On the other hand if he is there for longer periods and he is on the autism spectrum (or has other SN) then his needs may start to challenge them too. Still, could be worth a try.
Anyway, autism-spectrum or not I hope you find a way through Awaitingthedreamboat 