He is doing you a favor by being clear. Love your kids and pour that extra kid energy into developing your own interests.
This is what I came here to say. I was in exactly your position OP. Right from the start I wanted 3 kids and right from the start my DH didn't. He claimed to want to wait and see. After the second I kept asking when would be the right time. For years he knew I wanted a third child.
Eventually, as DC2 approached school age, I said to him that it was now or never. I was 37, I had a career that had been on hold for years, and I knew once DC2 was at school I wouldn't be able to go back to the baby days. I told him that if he really didn't want a third child I could get past this and move on ( not easily) but it was decision time. He immediately told me he didn't want another and that was that. He'd known for 4 years he didn't want another child and wouldn't even do me the courtesy of telling me that. He kept me hanging for years and another 4 years later I'm still quite resentful to be honest. He's always been too scared to have a difficult conversation about anything, I shouldn't be surprised that this was the same.
It was so hard, but I did get past it, and sometimes I am grateful that I don't have the extra chaos that a third child would have brought, my two plus my job sometimes feels like as much as I can cope with. I went back to full time working when they were both at school and quickly gained promotion, I feel very fulfilled by my kids and work now.
I'm still astonished by his selfishness every time I think about it though. His reasoning was that it would be too tiring and he wasn't up for more sleepless nights. He knew how much I hated pregnancy, I was so sick both times for the full 9 months. That it didn't occur to him to think how much I must have wanted it to volunteer to do that again really disappointed me.
Be glad that your DH is being up front, and try to focus on what you have. Is another baby worth tearing your family apart for? I agree with the others who have said that the partner who doesn't want a baby unfortunately trumps the one who does in this case. I feel for you though, it's really hard. 