Stop letting him bully you. The children are meant to be back at a certain time, nothing has changed and they should be back with you as previously arranged. Do not accept his changes to the arrangements just because he feels like keeping them to make himself feel better. Stand up for yourself and for the children!
Also there is no relationship to discuss or justify to him. And even if there were it is none of his business. You should tell him so in no uncertain terms.
I suggest you inform him the children are to be returned/collect as usual at [time/date] and send him an email or letter along the following lines:
Dear Ex
It is inappropriate for you to judge who and what relationships I have as we are no longer in a relationship other than that of parents to our children. Who my friends are is frankly none of your business, as it is none of my business who your friends are. I'm sure you would consider it inappropriate for me to comment on your friends and relationships at this time. Therefore I suggest you do not take it upon yourself to cast judgement or make comments on mine.
For the record the person described as my 'boyfriend' by [child] is, and has been, no more than a friend to me at this time. Also he has not been introduced to the children. I will decide if and when it is appropriate for that to happen.
If you have issues over who my friends are then you should speak to me and not involve the children.
It is completely inappropriate to threaten them and refuse to see them over a simple misunderstanding. I am sure when you calm down you will see how inappropriate your actions are and how it is not in the best interests of the children to threaten to refuse to see them any longer over any issue.
It is clear to me that you have issues which you need to resolve in regard to the situation between us and also as to your role as a parent to the children. Perhaps you should consider seeking some kind of professional help or councelling to help you resolve these issues. The children are suffering un-necessary distress through your actions and behaviour and I'm sure you will agree this is not in their best interests. I hope we can quickly resolve this situation for the sake of the children as it is not in their best interest to be put through a situation such as this.
Please be aware that I have reassured your mother that, as far as I am concerned, she will always be welcome and able to see the children, even if you choose not to see them yourself. Please do not imply otherwise to her again.
I do not feel there is any more to be said on this subject. However if you feel that anything further needs discussing please contact me directly.
Yours
[you]
Keep strong, don't allow yourself to be bullied like this. Your children need you to be strong for them given the completely inappropriate behavior and jealous actions displayed by their father. Also speak to your new man and explain the situation to him. If he runs then he's not the man for you anyhow.
Best wishes
Gilly