Ex partner interrogates our son about my new relationship - making us all miserable

[Tweetabix]

We separated 5 years ago, but my ex has never accepted the separation.
He has our DS to stay around once a week (at my request, ex never invites him or asks to see him).
I am now - after 5 years - in a happy healthy new relationship (for just over a year now). My son gets on with new partner really well. We have been taking things very slowly (it took me a long time to get over the split), and everything feels very positive now, and mature (compared with what I had before!).
Except for the fact that every time my ex has DS to stay, he interrogates him about my new partner. (Eg. Has he been to the house? Has he stayed the night? etc) This is making DS very uncomfortable. He wants to tell the truth to his dad, but has realised that the truth makes his dad angry or depressed. He feels torn between us. He got very upset the other day because ex told him that he had seen my new partner and think he is "very ugly" (he isn't!). DS has autism and high anxiety, and this sent him into an awful tummy-achey state. He feels the need to agree with his dad, out of loyalty to him. But also feels that would be disloyal to me.
I feel so so angry with my ex for making our DS feel this way (and this is just one example of many). I have tried to talk to my ex about it, but he always responds by saying "it was only a joke and he KNOWS it was" or "I'll say what I like to our DS". I try to explain how it's making DS feel, but he then accuses ME of being the one who makes DS feel bad, by causing our separation in the first place - it is all my fault, always... He says, "Well you know what to do about it" (meaning I should get back together with him and then everything would be alright...) I can't ever make him see how his words affect our DS. I feel so worried about it. DS is having trouble with sleeping and real big problems at school. This is partly due to his ASD but he has told me himself that a big part of his stress is his dad and the feeling of being in the middle.
I know that one solution would be for DS not to see his dad. But I feel this may be too drastic a step. DS does want to see him (though mainly because he has an Xbox!!) Also, the once-a-week night that DS stays with his Dad is my only chance to see my new partner on my own. If I don't get that reasonably regular time alone with him then I fear the relationship will suffer. (DS staying with his dad is the only way I can get a night off - there is no one else who can have him for the night).
Anyone else going through a similar thing? What to do?

Just to explain, my DS is 14. So his bed time is pretty late now. As I said, he wants to see his dad (I said mainly because of the X-box - but he does actually want to see him anyway!) He and his dad are currently bonding a bit for the first time (ever), because of their mutual interest in electronics, and DS is now old enough to engage with that with ex. In fact he still asks me to "allow Dad to come back and live with us". (I guess many kids just want their parents to be together, no matter what ). However, along with the fun times comes the interrogation, and the slagging of "Mum's boyfriend".
Being the single parent of an autistic child is an incredibly lonely and hard job, so yes of course I put my DS first and always have done, but I can't help feeling that I need the occasional bit of grown-up time myself, and that his dad needs to do his share. But then he never has done, even when we were together, so I guess I'm flogging a dead horse... I used to see a relationship counsellor who told me that my trouble was that I kept hoping for change, when there was clearly never going to be any! So I do accept that is my problem, always has been. However....I have decided that today is the day that I talk to him about it (again) but in firmer tones, probably using words along the lines given in NovembeR's message above.
I would like to help my son feel able to say something too, but that's tricky one. He may be 14 but emotionally he's very very much younger. His ASD makes him pretty vulnerable, emotionally, and my ex never seems to have got that.
Thanks all.

Not that it really matters but is your Ex on the autistic spectrum? It doesn't excuse his behaviour but may explain why he thinks all problems would be solved by him moving back in.
I have a child with autism. It's extremely hard and I'm not a lone parent. Especially as the teen years progress and our children are so different from typical teens.
The only thing I can suggest is repeating simple and straightforward responses to your son's comments. 'I am not going to stop seeing J, it is none of your dad's business' 'I don't want to live with dad' 'i can't stop dad being sad and angry and neither can you. That's not our job' and maybe give him some responses he can use with his dad?
These are just 'top of the head' suggestions. I feel for you and I hope things improve.

@Coseynightin

This is quite funny reading the responses if it's not what i like stop contact with the dad total control freaks

It is not controlling to not want her DS to be emotionally abused.

His dad questions him to the point where its causing him a lot of distress - do you think thats ok?

OP have you tried mediation? Sometimes a third party can get through to them when they don't want to hear it from you.

Thanks Spanglemum - very supportive post. Sounds like you have a good understanding of the situation! And yes I do suspect that my ex is also in the spectrum. As you say, it's not an excuse but it does explain a lot of why is the way he is.
SecretRedhead - I just wish and WISH that my ex would consider mediation. I have tried, believe me! I feel so strongly that he would listen to someone else if only he would give it a go. But he has refused on several occasions so I'm not sure it's worth trying any more. He refuses all forms of outside help (or "interference" as he would view it.).