Can you make his Sunday visits your time off? Stay home and let him take the baby for a couple of hours while you spend some time relaxing - don't clean or do jobs, just put your feet up.
He sounds very selfish and I'm not convinced he'll ever get better - having a new baby is such a massive change, but it's the one time that most dads will actually step up and give you some proper help because they understand that both physically and emotionally you've been through a lot.
I wouldn't advocate leaving or making any massive changes right now, because it's a hard time for everyone, but this is definitely him showing you who he is as a husband and dad. Which is a self centred unhelpful one who pisses off out for the night instead of giving you some much needed time off.
Separating isn't an easy option, especially with such a young baby, as you will end up doing everything yourself every day, but it sounds like you already do and at least if you split you might get a few hours off each week when he has access.
At your baby's age it's unlikely that he would get overnights for quite some time, but even if he had the baby for a few hours once or twice a week, the rest of the time you wouldn't have to put up with his sorry arse.
I'd be thinking of it as an option for the future if he doesn't sort his priorities out. xx