This could have been me a few years ago. Below is what hindsight has taught me, it took a long time to do so but it is what I've learnt over the last 5 years...
CSA is an entirely different topic and not related to visitation in any way, continue hounding them but don't stop contact because he won't pay.
Separate your feelings and the way he is towards you with how he is towards the child (providing there is no violence). Continue using a middle person for handover if it makes you feel better.
Continue to support contact at the Contact Centre. As your child develops, becomes more mature 'interesting' he might stick around more.
Providing there is no issues with violence and when he pushes for a different sort of contact move it to a supported stay and play type session at a Surestart centre.
After this an unsupervised session at a soft play area.
Build up slowly to unsupervised access anywhere (picking up from yours/parents -Do NOT do the running around) I increased this from 2hrs to 10-4 an hour every fort night)
If it gets that far with regular visits that he is demanding overnights, then put this in place from 3yrs later if you can manage it.
My court order was Wednesday and Sunday 10-4 to be slowly increased to Wednesday every week and every other weekend from the age of 3 10 sat-10sun.
Alternate birthdays and Christmases etc.
Go to Court it's not as scary as it sounds, really work on what you want your court order to look like, it's so much better to have it all written out. And be prepared for him to tell you your going to lose residency etc etc and don't rise to any of his insults just record everything.
Get the passport sorted, you don't need dads details but better for you to get it and look after it than him.
It takes a LONG time for the situation to clear, even now the only discussion we have at the doorstep is when he'll next see her, what she's eaten and whether she's been poorly etc other than that we parent separately and it works for us and DD doesn't know any differently