Sadly, you have a man who is a lazy-arse git who believes it's 'woman's work' to look after the baby. Many men think this, including the men of some women who are bashing your partner here, and even having a go at you for letting it happen. They won't admit that though. 
Mine was the same for a time, until I started going downstairs onto the couch to sleep - because of his snoring, and letting the baby cry in the room next to him. Most of the time I went up and saw to the baby, but occasionally I left the baby crying, until HE went to the bedroom to see why the baby was crying, and dealt with the issue. These men need to be trained, and to understand that no WAY is it just the woman's responsibility to look after the baby!
Men aren't generally stupid, awful, lazy lumps who don't give a shit about you and their baby, they just have it ingrained in them from an early age (usually from their mother who spoils them/placates them/makes the daughter(s) in the family do 'woman's work.') Or a misogynistic father. Men don't behave like this (leaving EVERYTHING to the woman,) for nothing. They have learned this behaviour, and they need educating.
You need to take the baby into the bedroom at least occasionally, and just leave the baby THERE, with him. And then go back to your bed! HE is the baby's parent too, and even if he works (more hours than you,) I am willing to bet that you do a lot more in the home, and with the baby.
He needs to do his share. No way in fucking hell would I have a daughter of mine going through this. I would give my son-in-law a short shrift and tell him to buck his ideas up! Same with a son. If he was being a dick like this, I would tell HIM to buck his ideas up too, and help with the baby more.
And do ignore the 'leave the bastard' cries. No woman/new mum would EVER do that. (Unless there was abuse.) As if leaving him would make your life easier! You would still be looking after the baby - even more probably - and having to deal with everything else, and having less income.
OP, you HAVE to talk to your partner, and he HAS to help more. I am willing to bet that you have said barely anything to him yet.
Show him this thread, and how you are feeling.