He has many good qualities as well but yes has said a lot of things that hurt me. It's a lot more complicated in real life than just LTB as is advised on here.
Would you still drink your cup of tea if it had a teaspoon of shit in it? After all, a cup of tea does have some good qualities, it's not ALL shit.
Do you think that many of those of us who advise to LTB don't know it's complicated? That we've never been through it ourselves? Of course we have! I remember thinking I couldn't really complain, because most of the time, he was just aggressive, and he sometimes slapped or pushed me, and "only" punched me a couple of times. Other women ended up in hospital, or dead, so it wasn't that bad really.
That thinking was really fucked up, but eventually, I realised I didn't have to put up with it, and I left him. And no, of course it wasn't easy, but my life was a damn sight better afterwards.
You can also leave, and you should leave. You should speak to Women's Aid. He may not have physically hurt you yet, but if he's threatening it, then he's thinking about it, and he's likely to become violent.