That man has got nerve and like I said, he's got an eye for his main chance and he will take it. He feels entitled to what you have and your boundaries mean nothing to him. He's a shameless parasite.
Send him an email and a text that he has no right of entry to your home and he's not allowed there at all. I hope you have doorbell and security cams. And then you should block him everywhere. Ask your neighbours if they'll keep an eye out and if they see his car, call the police, he is not supposed to be there.
Pithy, he's not going to leave his kids out for you. He wants what you have for himself and them. That's why he won't see you unless you agree he and they can move back in. He doesn't care about your welfare and all your pining for him won't change him. That's magical thinking.
Him asking for your keys shows it's time to cut the cord. Him getting mad and trying to guilt trip you over you saying no to such an unreasonable ask, this shows what his goal is and it's sure not showing any care and concern for you and your son. Those happy times are long gone. You can't live there. You need to look at right now, today.
He's trying to move in again and he sees your holiday as a chance to get his feet under your table again. He's greedy and manipulative.
Keep him away from your son. He's not good for him.