I'm struck by your description of how your friends find him. I knew a guy like this once. (incidentally, you/he don't live in the Northeast, do you?).
On the surface, this guy was an asshole. Nobody likes him. Clearly very intelligent, and raised with Olde Worlde manners, he could be charming, but he talked too much, to the exclusion of others getting in edgewise, etc. So he developed a bad rep amongst our group.
Once you got to know him a bit, he was nicer. He got more charming and inclusive. You felt you were lucky to be part of the Inner Circle of [name] and that felt special. It was easy to think he's just misunderstood: nobody sees The Real Him (like wot I can - I'm a bit of an acquired taste as well.) I was in a failing relationship, and needing to make my mental/emotional break.
I got to know him well enough to learn that, beneath that surface layer of Asshole, and that middle layer of Potential [to kid myself], there was a heart of solid-gold Asshole.
I didn't have Mumsnet in those days. Now I'd have the wisdom (even if only "on loan" from the wise women
) and the vocabulary to understand that he was just a selfish shit: he could be arsed to try only when he needed to. I might even apply words like abusive or narcissistic, although I didn't let him get that close to me that I can just pop out that diagnosis.
Your declarations of how you can't help but lurve him, well, sorry. That's not love. Deliberately choosing to be with somebody who makes you unhappy is self-flaggelation.
I do hope you wake up and enjoy your new life, your freedom.