OP, I can somewhat sympathise with you because I too am 23 and have an unplanned DD. Me and the dad are still together, which is a big difference, but I am in a similar situation regarding university. I found out I was pregnant a week before I was due to move to uni and start the exciting life of partying and self-discovery.
Firstly, I would say you should stop blaming your ex for the situation you are in. When you were pregnant, it was your decision to make, and whatever promises he made shouldn't have factored into your choice. You either have the baby because you want it, or you have an abortion. DP made a lot of promises and sacrifices for me, but truthfully it wouldn't have mattered anyway - I knew I wanted the baby and would never have felt right with a termination, so I kept her. When I first got pregnant, even though I was already engaged with DP and deeply in love, I had to ask myself: "If this all goes wrong and DP leaves me, will I be able to cope?" And I decided I could, or that I'd give it my very best shot at least. You can't blame anyone but yourself for the position you are in now.
Secondly, though my DD is only almost 12 months, I know how heartbreaking it can be to think she doesn't love me as much as she loves everyone else. She screams bloody murder when she comes home from her weekly visit to her grandparents. When DP first gets home, she clings on to him and cries whenever he hands her back to me. But I always know if it was the other way around, and I was the one who she only saw for a few hours a day/week, she would be the same for me. She just doesn't have to express her love for me, because she knows I'll always be there. If I leave the room, she knows I'll be back within minutes. I think your DS is just struggling with saying goodbye to his dad, who he knows he won't see again for a few days. He will adjust to it in time, I'm sure of it, just as I believe my DD will when she understands better that she can't have everyone she loves around her 24/7.
Thirdly, I completely understand the urge and desperation to reclaim the life you think you're missing out on - the life you always believed you were destined to have. I went back to college at 19, spent two very isolated years in a sixth form where everybody was 3 years younger than me (and thought I was a weirdo) JUST so I could get the big prize of university at the end of it. I got good grades in my A Levels, got a place at a brilliant university in a big city where I could fulfil my dreams of getting pissed every night and sleeping until whatever time I wanted, and it was snatched away from me. And it is a horrible thing to have to come to terms with. And it doesn't hit home until you're not pregnant anymore, or caring for a brand new special newborn anymore. As soon as the novelty and excitement of parenting wears off, you're bound to go through a period of mourning for what could have been. It's difficult but you have to accept it.
I think you may still have PND, or a different form of depression. When I had (yet to be realised) PND, I actually managed to convince myself and even DP that it would be for the best that I moved away to uni and came back at weekends to visit. Deep down, I knew I didn't want to even come back at weekends. I loved DD, I loved DP, but I was utterly miserable. It wasn't the real me talking. The real me would never have wanted to abandon my lovely little family in the attempt to live the life of a single girl. Luckily, I quickly came to realise that wasn't the life I planned. I didn't go through nine dreadful months of pregnancy and 3 days of labour so I could hand her over to someone else and forget about her for 3 years.
Think of it this way. The first few years are the hardest. You did those years. You went through the teething, the night terrors, the scary fevers, the tantrums, the dangerous crawling towards plug sockets. Why stop now?
I think that once you've accepted responsibility for your situation, and realised that your son does love you and is just a child who cries when he has to say goodbye to his dad, you will probably feel very differently about university and travelling. I think you may be in denial about a few things.
I agree with an above poster that, if you're serious about this, you should talk to a professional about it. And potentially visit your GP regarding depression as, in all honesty, I think you just want to run away and are perhaps making excuses for yourself whether you realise it or not.
Feel free to PM me if you want to talk, and sorry for the essay.