Your boundaries with him are flexible and he's testing them.
Sorry, but what you're doing is called intermittent reinforcement and it's the strongest way to reinforce behaviour.
He shows up without calling or you inviting him, but you go out and talk to him. That just rewarded him with your attention.
He shows up unannounced again with your breakfast and you spend time with him. Again, you just rewarded him with your time and attention for crossing boundaries.
He wants to come and have sex with you tonight because that's his next big plan to get back in your house. You're going to have to text him to go to his home if he shows up. No face to face time.
It won't stop until you make him stop. No reward of your time when he just shows up. That's his consequence for crossing the boundary that he can't rock on up to your house as he pleases. You text him to go home. That's it.
Your actions aren't congruent with, that is, matching your words.
A man can't help you heal PTSD, especially a damaged one. He sure can make it worse.