@Alyssah2
So sounds to me as if his parents are only too glad to foist him off on you. Says a lot about him, doesn't it? They've come up with some pretty stupid and self-serving 'good reasons' why he is now your problem, haven't they? Reasons that he can use as justification. Don't swallow that bilge water. You don't owe him a damned thing. And having a baby with him certainly doesn't make you responsible for him.
As far as you being a 'family unit', you really aren't. Just living in the same house doesn't make a family, especially when the relationship is so one-sided and unfair (to you). A family is a group of people (small or large) who mutually support each other physically, emotionally, and financially. And who have each other's best interests at heart. That's definitely NOT him. In fact, he's quite the opposite, isn't he?
As far as a 'family' being best for a child, sure, if it's a loving, healthy, and cooperative family. But a family where there is no love on either side and advantage-taking on one side and (valid) resentment the other is NOT a healthy nor a cooperative family. In cases like yours, it's better to simply coparent or even 'go it alone' if the other party cannot cooperatively coparent. Whether or not you realize it, having him there is stressful for you, probably more stressful than any 'help' he gives you. If he was gone, there would be less stress on you, you'd be able to find your parenting 'rhythm' and be fine.
You do realize, don't you, that any day now he's going to just show up at your door with the cat in his arms? Because it's easier to ask for forgiveness than permission. What are you prepared to do when that happens? Give it a 'trial run'?