Well you have told us a lot about what he thinks.
You have asked for opinions and had responses about what MN think (which are pretty unanimous I’d say)
But it’s what you think that matters OP. And the reason you are struggling is because you put the thoughts of others above your own.
The best advice is to listen to your own instincts, consider them carefully and then act decisively on them, with confidence. The more you do it the more confident you become. That’s how self-esteem is developed properly. It’s how you feel about yourself not about how others feel about you. Aim to know yourself.
I think you are on the right track to better self esteem when you talk about dumping him.
But on the wrong track when you look for ‘a glimmer of hope’ to keep you stuck in this self-destroying pattern.
I hate to be blunt, but … going backwards from a relationship to FWB, back to a relationship and suggesting FWB again when the arguments start …all with the same person who is still married to somebody who despises him is really very self-destroying OP. The exact opposite to improving your self esteem.
FWB is very damaging to somebody with low self esteem.
This ‘relationship’ is going nowhere good for you.
You really would feel the rewards of being single and free and not being used any longer by this man. Don’t wait for him to dump you again. Make yourself unavailable for his use.
So, ask yourself how his actions and conduct make you feel. What is your true instinct. What advice would you give to yourself? what sort of life do you want and deserve?
Act on that and do it with confidence like you mean it.
All the best to you for your future.