Typical behaviour from such vile people.
He's trying to control the narrative by arguing back and forth with you, stalling and going when he decides to go.
This is your home and your rules and he either respects that or he doesn't get to come in it again.
I know its easier said than done.
It may be an idea to write a plan down for him - that way he cannot say that he didn't know that, or didn't agree to that or any other twisting things they tend to do when it suits them.
For instance:
These are the times I can do for visiting your child (away from your home obviously).
These are the times that I can do within her existing structure.
If you are unable to make any of these times then text this number (a good friend maybe or even get a burner phone and tell him you have changed your number - at least then you can turn it off if he gets excessive with the texts and calls).
If you threaten or intimidate me in any way or treat me with less than the respect I deserve as the mother of your child then there will be consequences.
Our relationship is over but I am happy to work with you to ensure that our child has the opportunity to maintain a fulfilling relationship with her father.
Have as little contact with him as possible, protect your home - it is now your sanctuary and your safe space - he has no right to be in it, even to visit his child. It is not your fault or your problem if he has no money or a lovely house to take his child to.
Stand firm, set your boundaries - hope for the best but expect the worst is the motto and make your plans around that.
These men are vile creatures that do not think or act logically - keep your guard up at all times.
As for calling the police - that is not stupid! He was doing this to intimidate you - call them if he does anything like this again - get those reports on file, get them alerted to this horrible specimen of manhood - you could even file a report online now just to inform them of this situation and any potential future ones.
That is what I did as my first course of action - I too felt stupid for ringing 999 when it was 'just words and intimidation' and it set the ball rolling for me - I began to realise how awful he was/is, that I needed help and support. They were very good and helped show me how abusive he was.