Sometimes, although something might be more work, it's much less stress.
My sister did it, because it was the best thing for the baby and herself. She just told us her decision and said it strongly with her mind made up, and I think that helped a bit. If she had obviously been less sure of herself, she probably would have faced rubbish comments or not so helpful suggestions. Somebody always thinks they know better.
Stay firm in your decision. Don't let anybody else who isn't raising your child try and tell you what "should" be happening or how a home "should" be.
Just know though, that you will get judgment regardless. Literally any decision a parent makes, you'll have people agreeing and other people judging. If you stay with this man, one group of people will judge, if you leave, another group of people will judge.
So don't worry about other people too much, because there's nothing you can do that all of the people will agree with.
If he is rubbish, you wouldn't have had great support from him anyway, so would have been doing basically single parenting, just with a guy in the background.
There are pros to single parenting too btw. Loads. You get to decide what happens, you make all the household decisions and don't have to consider a partner, you decide what nursery/school the child goes to, you decide how the child will be raised at home, you decide what you want to do every day etc. Ask any single parent, it might be difficult at times, but there are still pros to it. Both have pros and cons, like anything really.
Just stick firm to your decision and keep supportive family and friends round.