YABU. But I can also understand why you felt slighted. I think the more important question is "Why did his off the cuff remark make me feel bad?"
I think that your assertion that you want to go to work for "you" is a valid aspiration. But many working parents aren't that self actualised by the job they do that pays the wages, altho I wouldn't want a partner to suggest I did a job that made me miserable. Also it's a totally valid point that it's difficult to find part time work once you have been out of the workplace, and that it's daunting to get back to work when you have been out of the workplace. But you don't actually say that you've looked for a job, nor have you even discussed the issue between you.
I do think there's another discussion to be had when LOs start school. I also think that you seem resentful that his career has not been interrupted when yours has, despite you saying that you going back to work would have been "benefit neutral" - I can only surmise from that comment that at the time you would rather have stayed at home with the kids than work, so I would struggle to see why you are resentful.
Personally I went back to work when DD was 1, solely to keep the job that I valued and had worked bloody hard for all my working life. I worked 3 days which I would never have been able to do if I hadn't already been in the job. You can't change anything now and I'm not beating you up about it but try to remember the reasons you decided not to return to work - they were valid then and they are valid now.
You need to discuss with your DH whether he meant anything by his comment. If so, then I agree with Bogey that if the domestic chores are not shared equally a discussion needs to be had around that. You also need to look yourself in the mirror and remember that you have a raft of skills, many of which will have been honed over the last few years running a busy household (project management skills? coaching skills? organisation and administration skills? diary management?).
Ultimately if you don't want to get a job and your DH is happy with that, then that's nobody else's business, you just have to accept that the bigger house won't be yours (which I am sure you do accept, I realise it was just a passing comment).
Did you stop to think that maybe he was offended by your (perceived) wish to live in a larger house which he is currently unable to provide?