Thank you for the replies and suggestions, I'd given up so I am really glad people replied.
He was a very unhappy baby, at least at first and he cried solidly for the first month or so. I kept asking what might be wrong with him as it was so painful to listen to, and I was doing all I could, carrying him instead of putting him down, often while pushing the rather redundant pram, feeding on demand, all that stuff.
No one seemed to know what was wrong with him. They just said 'it's normal, babies cry'. Well my middle one didn't cry but now we know he probably has an ASD, but even so, even my first didn't cry on that level and I did most things the same.
He hasn't had another hearing check since he was born, he often whispers and can understand whispering as well as normal talking so I don't think hearing is likely to be an issue?
I always hope the fact I'm not shouting at him will make people less judgmental. It's mainly me carrying him while he kicks and screams.
Thank you for all the tips about talking to him; I have tried quite a lot of that, the choices thing and so on. It worked for a few months but now if he is cross he is just cross, and sometimes furious, and nothing works. If I try to walk to the kitchen he will scream and grab my legs, and when we go out he will actually climb out of the pushchair harness (if I've managed to get him into it at all that is, which is highly unlikely) and just run off.
My mum often says things like that, as though she is rejecting the fact that she might just not be able to 'baby whisper' this one, you know, it must mean there is something wrong with him, he is faulty.
She is often very helpful in practical ways and is kind. But she says such stupid hurtful things too. She has done it before; my second one wasn't normal either, according to her, and she didn't hold back on that topic, and it was obvious she struggled to love him. Now she can'tremember saying any of it 
But anyway I just have to try and ignore it and I tell her it is hurtful and she normally apologises.
I mean it isn't like I don't already wonder if he is OK. Of course I do.
In his 'normal' times he is lovely, sweet, talkative, wants to play with me, or his brothers, concentrates, draws pictures, laughs a lot. He likes physical play and to be 'thrown around' by his biggest brother. (gently!)
He doesn't go to nursery and tbh I don't think I could find a nursery that would manage him, the way he is now. I also wonder if it would make him a lot worse if part of it might be separation anxiety, or similar.
He does his best but he is really struggling with something, and I don't know what.