Well done OP. You are taking the first steps.
The issue here is what is reasonable in terms of you facilitating contact. He is getting in your head by suggesting that unless you do what works best for him, you are being obstructive. This is a fallacy and he is using it against you.
Rather, as others have said, you can't obstruct him... from reasonable contact. So no, you can't refuse to make them available, nor can you chop and change your agreed schedule. But if he does not have facilities to have them overnight, you do not need to have him in your house overnight instead. X1000 considering his previous abuse and that he does not actually DO anything.
At a practical level, you need to reiterate that you are happy for him to spend time with them, without agreeing to facilitate that. "I can make sure the boys are free on Saturday and/or Sunday and am happy to drop them off. Let me know.".
Then grey rock the accusations etc.
Re court.... hahahahahaha.
1 he won't be bothered. He can't even be bothered to look after his children, he's not going to court.
2 he MIGHT get a solicitor to send you threatening letters. They are just that - letters - until you get formal notification from the court you don't have to do anything. If he does write to you, a calm factual response (once) sent to his lawyer is sufficient
3 in the unlikely event he does take you to court, there's very little he could win. While the courts often get it wrong, I am 90% sure that no court is going to order you to let him into your home, and certainly not into your bed.
4 if he does take you to court, you have evidence of his behaviour - previous abuse and current harassment in person and via text. Carers' concerns re his behaviour (and I would see what, if any, formal evidence of this you can get), his inability to create a safe and appropriate space for the dc in his home etc.
I think i may have warned you in an earlier post that he will rant and accuse you of things. But I will remind you of the other point I keep making - he is messing with your head, but if you look at every decision you take and take an honest, and rational, view of whether it is reasonable, then hold that knowledge in the front of your mind when you have to engage with him, you will be on track.
Also, I have to say that everything you have said about this man tells me that he does not care about the children. His behaviour is ALL about getting his "supply" from you. Once he cannot get that, he will start by spiralling out of control and ramping up his behaviour (ie things will get worse before they get better), but then he will most likely disappear completely.
Look up covert narcissism.
Good luck!!! You are doing so well.