Look, what no lawyer will tell you is that you don't have a court order in place, therefore you wouldn't be in breach of a potential order, however what you need to decide is whether or not you would feel safe in a country where you don't have family with an ex who has been violent. Would you be constantly looking over your shoulder?
You could get supervised contact which is more secure than supported contact at a contact center. Supervised contact means you don't come into contact with your ex, but also there is someone observing contact, and someone else who stays with you and someone else who runs between, for instance to take the child to the non resident parent and back after a visit.
LIke you say, he is probably doing this to exert some control over you, but in reality he will show up for a contact session, find out what is involved, namely that he won't see you, and his game will be over. He will probably lose interest on seeing DD.
But if you go to Ireland, you won't have much of a support system, and may feel very isolated.
Your solicitor could ask for supervised contact to be a criteria for granting contact, but emphasize that you want supervised contact, and not in a contact center, but in a designated children's center.