5 years or so from now when you look back I can guarantee one thing. You will regret the time you wasted worrying over this idiot and not bonding with your daughter. Believe me I know only to well from bitter personal experience where you are. I can only tell you what I wish someone could have told me.
The world in your head is not the real truth of the world.
This man is NOT the only man for you.
Right now, your feelings are telling you that you only want him, that you can't live without him, that there is no other man out there for you, that he is your child's father and you are bonded to him for life. That because he is your child's father you will love him always. That you could be a family. And on and on.
Firstly, what you are feeling is normal. It isn't a 'fantasy' to want to be with the man you have a baby with. Its a normal, even healthy, desire. Something most people grow up wanting.
The thing is the situation has gone wrong. And you must acknowledge this. And although you can have the dream family you've always wanted, It will have to happen some other way, further in the future.
But it will never happen if you keep listening to your feelings and going back to this man sexually or otherwise. People say 'follow your heart,' but our emotions can tell us lies. This is because emotions take longer to disassociate themselves form any sexual or emotional partnership. Your feelings are lying to you right now.
You are lovable and can have a lovely family in the future. At 23 you have many years to get your life back on track. Would your mum support you while you work towards a new hobby or career? Uni? Think about the opportunities that are out there away from just thinking about this immature idiot.
Not contact is the best way to move forward. Cut down contact to bare minimum. Arrange pick up arrangements through your mum.
My advice is to pick a new path and focus on it like it is the holy grail. That's why I suggest maybe a new career. Is there anything you always wanted to do/be that you haven't achieved yet? Travel? Doing things helps the mind to travel new paths away from this man. One child as a Single Parent with some support is still flexible and you can achieve a wonderful life which includes marriage and family.
In the meantime if you need more help talk to your health visitor and go back to your doctor.
The future has huge potential. Take one day at a time.