Ok, so your update is really very unsurprising.
He's behind on maintenance anyway and uses it to control you? That makes it easier in a way, as that threat has already been spent, so to speak.
Stop contact, and send him a solicitor's letter telling him that from now on, it will be every other weekend with the option of an overnight in the week. That's the norm, and it's the norm for a very good reason - it's in the interests of the child to have weekends with both parents. It shouldn't cost much. You could ask for advice on listing in it his behaviour so far with threats around pickup etc. which also show that his current behaviour is not in her best interests.
If he doesn't agree, he can go to court. Simple.
And he won't... because as soon as he sees a solicitor, they'll tell him he has two options - accept the revision, as it's the norm and he won't get every weekend, or, go for 50/50, which he might well get, but he'd then have to put her before his work and work around having childcare, evenings in the week etc.
He'll go for your every other weekend. Perhaps with a lot of abuse thrown in, which you can then call him up on, legally if you have to, again with the threat of him not getting her at all if he's going to be abusive. And hopefully he will pipe down. And you can go to CSA for maintenance.
Do this NOW, because at the moment she is young enough for it to go over her head if she doesn't see daddy for a few weeks. It NEEDS to happen for her good - and his, though the twat doesn't realise it - all he's going to end up with is a resentful little girl who hates the way Daddy is forever whisking her away from fun, friends, home - always on his schedule never hers, always in his interests, never hers. He's doing it already - dominating, stamping over what's good for her because what matters to him is what's good for HIM. She'll grow to hate him - no point in telling him that though, he won't want to listen.
Do it for her sake. Take advice anyway - family lawyer should be able to set it out for you. Oh and don't be afraid of him - he really has no power here, which is why he bullies.