As said by me and countless others upthread, this man is a rapist. At best he is a sexually coercive but based on what you've said he is a rapist.
This specific relationship aside, on a previous thread of yours about the dynamic of being an OW, I shared the following and wondered if it's worth considering as regards your low self esteem and feeling trapped by this relationship when it's never ever going to be easier to leave him than now. He's already trying to stealthily move in. Here's what I said then:
You say you hate that this is how you are.
So what have you actively done to work on changing it?
Have you had counselling?
One of my friends was like this in our early twenties and counselling helped her realise that she felt validated and very attractive if someone was willing to risk their relationship for her. Which is all kinds of fucked up but meant she could work on it. And realised that most of the time they don't leave their wife, so they are willing to risk the wife finding out, but not willing to actually have a relationship with her. So what she felt was validation was actually her being viewed as someone worthy of sex but not love.
She isn't a very nice person and we aren't friends any more for a variety of reasons but to her credit once she uncovered that in counselling and worked on it, she did stop having relationships with unavailable men.
Could that underlying reason be the same for you?