I signed up to mumsnet just to respond to your post. Your case is very similar to mine and I have the benefit of hindsight (7 years of problems, 3 in court).
First of all, as the primary carer of the child, it’s perfectly within your rights to refuse unsupervised contact if you think it is unsafe for the child. The father’s lack of engagement and familiarity with the child’s needs are valid reasons for concern.
That said, some kind of contact arrangements will need to be made and, as a previous poster said, you need to set clear boundaries now.
I wouldn’t worry about him taking you to court now – court requires money, time and effort, which he doesn’t seem willing to give. What worries me is his threat to take you to court, as he might continue to use it in the future to make you agree to any other demands he has. Your child is only 10 months old, this could go on for the next 15 years if you don’t stop it now.
I think you should show him you are not afraid of court, but you also need to prepare for possibly going to court in the future. Keep as many communications as possible in writing (texts and emails are ok). Always be civil and composed and don’t engage in arguments. Make notes of dates/times he was late or didn’t show up.
Most importantly, make a reasonable plan of how you propose to build up contact and how future permanent contact arrangements should be. Include a timeline and clear rules (e.g. if he is late for x minutes, the visit is called off) and send it to him in writing.
Do not negotiate for too long. If your proposals remain too different after a couple of discussions, or if he is unwilling to negotiate, initiate mediation. If mediation doesn’t work, go to court and get a child arrangements order.
I hope you know that without a court order saying that the child lives with you, the father has exactly the same rights as you, e.g. the police won’t help you if he refuses to return the child after a visit.
I’m not saying it will get to that point – as other posters have said, he might lose interest. I’m saying that if he’s being difficult and refuses to cooperate, there is a good chance he will continue to do so in the future, and you should take steps to protect your child and yourself.
About CMS: you have every right to require that he pays his fair share for his child. However, you should be aware that many court applications by fathers demanding 50-50 shared care (which is usually granted) are triggered by the mother’s CMS application, as 50-50 means the father needs to pay nothing. I have no advice on this, I just think you should know it.
Sorry for the long post, I hope some of this helped. I wish you the best of luck.