I wold leave in your shoes, and did leave in mine.
My EXDP was not physically abusive but emotionally battered me until I had no choice but to leave.
He'd say nasty things in front of the DC about me, and always treated me as someone to look down on. He showed me no care in the way I needed to be cared for, and always put himself as number 1.
He didn't do any of the grunt work for either DC, particularly DC with additional needs, yet when I said I'd leave, he'd threaten me with taking full custody of the DC because I was such a bad mum and apparently did nothing in the house!
Since leaving him, he had begrudgingly had DC every other weekend, so 4 days in a month. He cancels visits if he decides he'd prefer to go abroad, and sometimes collects them on Saturday instead of Friday from school, at the last minute.
He has them a couple of extra nights here and there in the school holidays.
I'm currently in the process of taking EXDP to court, so we can put a child arrangements order in place, as I cannot stand the way he sees parent as optional for him. By the time the process is finished, it will have cost me the best part of £5k.
If I had my life again, I would always have left him (and probably not have continued the relationship beyond the dating stage early on). As other PP have said I don't want my DC thinking that kind of relationship is normal, or being the abuser or abused in future. I hope to be building a strong enough relationship with my DC that while I appreciate that they love their Dad, they'll always know I tried to do the best by them, while they'll come to see that he only ever thinks of himself (there are some signs he's emotionally abusive to DC, but not a lot I can do about it).