Decades ago I did a creative degree where we were put into groups of 4 to work on a project. I was 24 at the time and had been with my DP for 4 years.
He became fixated on a young man in the group who had a passing resemblance to him and suspected that I was having an affair.
I was not. And I must stress this as it would be pointless of my to write any of this unless I am being 100% honest. It struck my friends and I as absolutely bizarre at the time, but it got to a point where he was checking constantly and once even appeared at a house we were working in and threatened to knock the door down. We broke up over it and I moved out into a friend's house, where he was convinced I was still conducting an affair with this person. The guy in question was nervous but persisted in the project on the advice of friends, but was willing to call the police if anything like that occurred again.
Fast forward 18 months or so - we eventually got back together. Yes I know this was stupid of me. But nothing like this ever happened again, and we seemed to grow into a decent relationship afterwards and put it behind us. Over the years I have been terribly unhappy, and only have myself to blame for sticking around. He has never fully committed to me as a partner so we never had children, bought a house together or any of the usual trappings of a big relationship.
I am fine about the kids and house, as those were never a strong urge for me, but I do now regret not leaving sooner. All of this time he insisted we were great and would live together eventually.
But as it turns out he always looked after himself, with only an illusion of including me. As the years have gone by he has withdrawn from me more and more, doing most things alone or not even including me in plans. He insists that he does and that I am reading him wrong 
Whenever there is an argument he will occasionally mention the 'affair' that I never had and says it ruined his life and sexuality. He has not had intimacy with me, or anyone else, for 7 years now. He squarely blames this young guy from decades ago. Yes, it's just mindboggling. He acts as if no time has gone by and still has deep resentment. Then he will shove it aside and get on as normal. I am increasingly uncomfortable with this.
My problem is, because he is mostly quiet and nice, it feels difficult to get my head around it. Every bone in my body says leave, I have wasted enough life on this man, but he denies creating distance between us, even as I see it happening. There have never been any threats, or verbal/physical abuse.
Where we currently live is absolutely awful and I have wanted us to move for many years. He has excuse after excuse as to why he can't join me, but I am ready to get out. I am at my wits end with feeling lonely, even though he is always physical there. He has refused to come on holidays I have booked for one reason or another, and it often leaves me feeling that my mind is falling apart.
I know that I am stupid for putting up with this, and I know that I don't need permission to leave.
The last time we spoke about the lack of intimacy, he insisted he was madly in love with me and sexually attracted to me, and always had been. So I asked him why we lived this way, and he replied that due to the guy from decades ago, he could not fix the problem. I said any normal man would seek help for it, but he said it was impossible.
Along with so many other things that would take forever to write down, he is barely like a romantic partner at all nowadays, but insists everything is fine. WTF?