That is difficult, I understand that. Give what your ex said to you about 'telling your daughter he was dead', I think that you should try to think of some semblance of truth that will not hurt her. She'll join up the dots later.
What about, "DD, I know that you haven't seen daddy for a little while and that you miss him. I'm sure he misses you too, he loves you, but he is a bit sad at the moment and busy sorting out his things so he can't see you as often as he would like to. The thing is, mummy and daddy are not living together anymore but we both still love you very, very much and always will. If daddy rings me and asks to see you or speak to you then we can make some arrangements for you to see him. In the meantime, just remember that both daddy and mummy and x,y,z sibling, grandparent, aunt/uncle, etc. love you very much".
I seem to remember your daughter is 4; she should be reassured that all is well, she doesn't need to know the ins and outs of anything.
If you really think she is missing him, could you give her something that she can talk to/cuddle up with and tell her it's from him? It's a slight of hand but she's 4. Will she be a flower girl at your son's wedding? Could you distract her with that?
I don't think you should talk to him, he's been clear - and it messes you up too badly as you still have (understandable) feelings. This is about your little daughter now and what you can say to salve her feelings in a way that will make her understand that she can't see daddy now but that he does love her.
Does she have grandparents from either side, uncles/aunts, cousins? Anybody to distract her and keep her sense of family intact?