You're wired differently.
For you, sexual attraction includes attraction to personality and behaviour. His initial shitty behaviour was a turn-off, there wasn't any communication about this from either of you, and as such, every subsequence advance has made you retreat further and further to the point where he's been getting more frustrated and you've been feeling even more repulsed.
The trouble is, when in this situation, most men can't understand that they need to treat you better in order to get more sex, and it's a long process. You would need him to step up as a father and to be nice to you without expecting anything physical in exchange, and because he's been so useless, it would take him a very long time to learn how to be a decent parent and partner. That length of time may be unacceptable to him. You might not even be attracted to him by the end of it, because the damage might have already been done for you.
So... it's not impossible to find a way back. It's just very unlikely.
It's not uncommon to fall for someone based on their personality. When they stop making you laugh, when they stop doing little nice things for you like bringing you cups of tea, when they actively start making your life harder... then the only thing left is looks, and a large proportion of the general population are average at best.
If you observe the old, happily married couples in their 90s, the thing that keeps them together is the sense of companionship and support. That comes down to personality/behaviour, rather than looks. Looks bring people together, but behaviour is what keeps them together. A lot of men don't seem to get this. The old saying of 'a woman wants a man who can make her laugh' has a lot of truth in it - as long as she keeps laughing, she's happy. People want to be happy, and when you're really happy, you see things as being better than they really are.
Clearly a lot of men think they're God's gift to women and can get by on their dazzling looks alone and don't have to work at staying attractive by treating their partners with respect...
It's a long road back. I know there are reasons why you don't want to leave, but you have to consider whether you think you're both capable of taking that road back, on top of whether you actually want to try. You might not want to be married to him right now, let alone sleep with him. But do you want to want to be married to him and to want to sleep with him? That's the commitment being asked of you right now. If you want to want, perhaps you can agree to counselling. If you want to run away and not look back, you know what you have to do.