"I've said this before: All responsibility, no rights is what mothers get"
Utter cobblers. You can't force ANY parent, male or female, to act as one if they refuse. Sad, but true - how could you? And how would such enforced contact be good for the child, anyway? And any parent who wants contact with their child is, and should be, able to enforce that if the other parent is blocking it without bloody good reason. It is not usually one and the same parent doing both (I accept some are just being arses, but others actually love their children and just want to see them).
A dear friend worked freelance and shared care of the toddler DD. Ex took kid in midnight flit with a new bloke nobody knew about, and friend had to go via solicitors and Lord knows what to establish where they even were. Then had to mediate, because you do before you can hit court unless you want the judge to go 
, even when you know one side is using it purely as a delaying tactic. Then first court app't the ex had to be lent on to agree to any contact at all, and was only prevailed upon because the judge applied massive pressure. She offered a few hours weekly in a sodding contact centre. The alternative was a prelim. contact hearing which might have guaranteed more contact but wouldn't happen for another 4 months and would cost yet more money (friend not eligible for legal aid. This whole thing has cost him £££££). There were excuses at least half the contact sessions (illness, holidays, traffic delays) cutting into the time. By the time it was heard in full, he'd hardly seen his child for the best part of a year and he was given every other w/e and 1 weeknight. But she is playing silly buggers about keeping to those now, so back to court he is having to go, and he knows full well that only when a judge threatens to flip residence will she abide by the court order. This is a guy who was a fully shared-care parent, but due to her having sole care for the almost-year run-up to a court date, he's now week-end dad, without even getting many of his on-paper weekends.
Some people are arseholes and use their kids as weapons/treat them as possessions. Gender doesn't dictate that. We always knew his ex was vile, never understood how someone so lovely could be with such a cow, but he used to go on about her tough childhood as if that explained her being consistently nasty to him. And you can't sit a grown adult down and say "this wo/man is poison, do not have kids with him/her".
I may add that my father is a total creep and behaved appallingly to my mother (and all his kids, and my stepmothers, plural) so I understand your anger. But women are just as capable of treating their kids like dirt when wanting to hurt an ex, and in point of fact the adult with possession of the child in the run-up to a court case has almost all the cards, because the court usually sticks to the status quo. Just as many men are primary earners and have a lot of power there, the kids are, sadly, a source of power and many women do use them as such. Denying that is as pointless as denying a lot of fathers are crap and drop their poor kids post-split.
OP I think you're getting an unfairly hard time. I think your ex sounds awful. But sadly I think your kids need you to manage his awfulness and bend over backwards, as long as you can bear to, at least until they're old enough to work out what a waste of space he is for themselves. Because they need to at least know their father is part of their lives and cares for them to an extent. Is it possible that you suggest to ex that you communicate via email instead of phone/text? That way you can perhaps be a bit calmer, both of you... and it will also be clearer to any court (should it ever reach that point) who is messing things around, with a consistent logged record.