If he isn't doing a lot, I wonder what he is doing? Is he putting time into hobbies, watching TV, going to the gym? Because that would grate on me too. It's all hands on deck when a baby is born.
Even though my husband of very helpful, I didn't want him to have a long paternity leave because although it can be harder work on your own, it's also easier in ways to get on with it on your own. Even with my helpful husband, I still found myself feeling irritated and resentful at times because even with his help I was still struggling and I wanted more from him, which is ridiculous I know but that's how I felt.
If you talk to him, try and focus on how you are feeling, rather than what he's not doing as a first attempt so he's less likely to feel attacked.
If it was me, I would honestly be asking him to go back to work, and just say its too disruptive, you need to start getting into your own routine, and it can put a lot of pressure on your relationship being round each other like that all the time.
Do you have any family nearby who could help as well?