I've been through what you are going through. I had small children at the time, so leaving an abusive partner (husband), whilst of course the obvious, and only solution, was horrible.
I understand entirely how you look at the 95% that is fine. (Is it really?) and measure it against the 5% bad. But I imagine that there are many days in the month when there is part of you that is already tense and waiting to see if this is the day that he comes home drunk and abusive. And worried that if this is the day, maybe frightened that the abuse will escalate: because unfortunately it will escalate and become more frequent.
You say: "I am a bit worried that he will totally lose it and perhaps physically hurt me one day." He is ALREADY hurting you, and it sounds like you are already living in fear.
It's easy for everyone to say: "just leave" and whilst of course it's the obvious answer, I can understand why you haven't already. I understand your attempts to rationalise your partner's behaviour. I found it difficult to to acknowledge i had chosen, and love(d) a partner who was abusive. It's emotionally complex.
But, he has made you the repository of his inadequacies. This pattern of behaviour towards you will continue, until you interrupt it - by leaving. (He will only accept he has a problem when he's ready and you can't help him with that).
I don't know if you would call your partner 'charming' at times. I was married to a charming man. Life and soul of the party. But there came a point when he was not that man at home. I'd recommend an excellent book, which any woman who has a controlling partner would benefit from:
Power & Control: Why charming men make dangerous lovers by Sandra Horley
Make a plan to leave. Set a date, which could be very soon.
Talk to people you trust and/or Refuge type helpline
They will want to support you and help you FEEL SAFE
Decide who you can stay with for the next few days/weeks -
and leave.
Just imagine being able to breathe easily again, without a knot in your stomach.
If you feel you need some courage to leave, then find someone to hold your hand and BE BRAVE.
Take care. You can be free and safe again.