Hi OP, I am so sorry this vile man is behaving in this way and you must stop it now in a safe way. Is he physically abusive or aggressive or violent? Because he sounds abusive by words.
Yes, it is called Gaslighting en.wikipedia.org/wiki/Gaslighting
He is being vile and making life terrible and convincing you, or trying to, that you are to blame. YOU ARE NOT TO BLAME.
You said ... is there anything else I can do even if it's little sayings to tell myself every time I feel myself caving?
Tell yourself 'This is for me and this is for my daughter'. When you fly by plane they say to you if the plane starts to lose pressure you put on your oxygen mask before you help your child. That is because if you pass out (or cave in) you cannot help you or your child. So you must stay strong for you and for your child.
Mentally think to yourself 'I won't cave in because I am worth so much more than this and so is my child.'
If he is not paying any maintenance he is not father of the year. You have woken up after months asleep. Stretch and get up and off you go, making plans for your future for you and your child and each time you feel the need to cave in mentally stretch and get on with being a strong woman.
I agree with BitterAndOnlySlightlyTwisted they have put it much better than I could.
I also agree with AliceinWinterWonderland. I think if you must have contact then a safe, neutral public place is best. I also think spending time in a soft play place might end up being quite boring for him and he may eventually not be so bothered about contact so often. This may sound harsh but to me it sounds like he gets more out of being rude and abusive to you than seeing his daughter.
No you do not need to endure this forever, you can stop allowing him into your home on such a regular basis. You can ensure when you are together that he speaks civilly to you. If he is rude I would ignore him. If he is aggressive or abusive I would report him to the police for harassment.
And agree with Twinklebells totally when they say If he is abusive and she is in danger it will need to be a contact centre.
You said He will fight and fight this and accuse me of being 'evil'. I suggest you make a note of as many of the horrible things he has said and when that happened, (if you can remember). Plus any record of what has happened, whether he has been aggressive either physically or verbally, when he has fed her any inappropriate food or done anything not suitable.
I would seek some legal advice. I would not ever let this man take my child abroad.
Please visit this website and download the relevant pdf.
www.gov.uk/government/publications/international-parental-child-abduction