'Things being too raw'
That was your cue to rescue him. 
Back on topic, (you, your wellbeing, your happiness) we do tend to play out what our early experiences were.
If a parent was emotionally detached, we can key into that, and accept those emotional crumbs of relating.
Trying hard to get things 'right', please people, to prove our worth.
When someone puts us down or rejects us, we look to ourselves, 'what did i do wrong?' when sometimes, we have not done wrong but are partnering an unsuitable person.
Its not our job to reparent, mother, and make projects of these men.
Better we work on ourselves.
These men probably come from their own disfunctional backgrounds, and key into ours. Etc etc.
And the cycle goes on.
Until someone does something different.
You have done really well to get to this stage.
Realise that the anxiety you feel may be emotional triggers from all the past stuff, and nothing to do with him.
Just that he triggered it.
Im no therapist, these are just bits that have made sense to me on my own journey.
I hope its of some use to you.
If you do feel these 'triggers' of rejection and abandonment, regulate yourself with deep breathing, hug a pillow, journal, do a meditation, be kind to yourself and know that it will pass.
Sitting with ourselves is a necessary part of the process.
Get aquainted with the pain.
The healing is at the end of that.
