OP he sounds really disrespectful and unsupportive. And you deserve respect and support at the very least. The 'well, I work' comment makes it crystal clear how little respect he has for the mother of his children and what it requires of you. I'm sure you are absolutely exhausted, physically and mentally and he has absolutely no intention of trying to understand how you feel.
I think most couples that are able to make it work when they have young kids, absolutely have to work as a team. Whether one is a SAHP or not, the crux is that they both contribute to family life around the working ones hours.
In couples where the SHAP gets shafted it's because the other expects them to carry the weight of family life on their shoulder, and wants them to believe they should be grateful for this honour because they are having a lovely time 'not being at work'.
When I was on maternity leave, partner was up at more or less the same time I was - doing useful things to contribute towards family life. Either occupying the babies, doing laundry, tidying etc. That's not to say I didn't do more of it when he was out at work, of course I did because I was physically at home and he wasn't, but when they are at home both parents should PARENT.
We absolutely both do have lie-ins and share the load, this is because we want to support the other one because we're both exhausted. He'll take them out for a couple of hours on a weekend, and I'll do what I want. Equally he goes out with mates and plays sport.
I really hope you are able to talk some sense into him - not least before your little ones start adopting his ideas and copying the level of respect he shows you! I'd start by communicating with him and working out a plan for sharing the time around when he is at work.
xxx