Firstly, go easy on yourself. Looking after a baby is relentless. You should not be doing this by yourself.
DH and I have always done shifts at night - DH did until 12, then I'd do any further wakings and get up with him. (I do appreciate this is harder if he's waking for a feed). We each have a lie-in/rest day at the weekend, and no way would DH be spending 6 hours of his 2 days at home out with friends. He's a father now!
He takes the baby to let you eat/shower?!? How gracious of him
You need to sit down with him and explain the reality of the situation. 'DP, I know you work hard and are tired, but this is what life is like for me I am even jealous that you get to go to the toilet by yourself. We need to work out how I can get some rest before I become ill'.
I'll say it again - looking after a baby is relentless. The good news is, very shortly that little bundle of responsibility will start giggling for you, and giving you cheeky smiles whilst causing mischief, and it becomes less of a chore and more of a joy to spend time with them. Although at 7 months DS hit a stage where he'd whinge from waking for his afternoon nap at 3.30-ish until bedtime at 6-ish. We went on lots of pointless walks, and he went from 2 baths a week to at least 4.
If someone offers to take him for a couple of hours - let them! It's not about not being able to cope - you can and you are. But take some time for yourself. The more rested you are, the better mum you will be. It's hard to enjoy your baby when you're sleep deprived!
As for those family members making comments about your house - if you ever let them in again, hand them the hoover and point them in the direction of the laundry mountain and washing machine. Cheeky bleeders.