No yanbu, and no you are not being selfish to have your baby, with or without a man.
Just because your hope for the future has turned out differently again, doesn't mean you won't have a wonderful child like your first experience.
If I knew you in real life I would have deep respect for you keeping the child you want to have, even if you have (understandable) fears about it. In some ways, I would have thought it would be easier for you to do it all again, knowing you survived the first time round?
Once people who know you (like parents) see your little baby they will be drawn to the life in your arms more than thinking about how the life came to be.
My Mum was silent when I announced my pregnancy, because I wasn't married. But she now dotes on her grandchild and it just isn't an issue at all. It never comes up in conversation and she would be gutted to be without him. I think people will be the same for you too. But even if they're not, it's their problem, it's their selfishness and poor judgement, not yours.
Have your little baby and hold your head up with pride. 