No, he's not right in terms of being nasty, taking advantage of your being at home (i.e. leaving clothes on the floor like a spoilt brat) and putting you down. It doesn't sound as if money is the real issue, I too think he sounds envious. But I think what you want is more important than being pushed into something by him and as he's behaving the way he is then no wonder you feel unhappy atm.
Do you want to work? If you do, could you look at the kind of jobs you could get and then talk to him about it from there, pointing out the differences it would make to your lifestyles - i.e. xyz job (your old full time one) would be realistic and would increase your joint income by £x but childcare costs (after school etc, school holidays)of £y would have to come out of your JOINT family income. Work it all out as accurately as possible. Ditto with a part time job, work out what you would bring home and definitely point out what you would NOT be doing were you to be working. For example, if you were to work while your children are at school then you will NOT be able to clean, shop, cook etc etc in the same way you do now. You would need him to pull his weight too.
I think in your position I'd be cross at the implication that my contribution was worthless and would be trying to financially quantify the work I do at home - i.e. x hours/week cleaning and housekeeping, x hours childcare, x hours cooking. I'd probably work it all out at the going rate and present it to him with a 'if I wasn't here, this is what you'd have to pay someone for my services'. And that's not including sex (Joking) Sorry, maybe that isn't helpful but I do think that sometimes men have no idea what women at home with children do and its value. I recently worked out that if I were to go back to work once this one is born in November, the first £13k (gross) of my salary would go on childcare costs for 1 x baby at full time nursery and 1 x 6yo, after school and holidays only - i.e the cost is £9k or something net and one of mine will be at school most of the day! In our case dp would be happy to be a SAHD but I would recognise that his contribution is worth at least £13k gross based on the above! More if he does anything around the house too, although I couldn't expect him to, having bleated on for years about how being a SAHM shouldn't mean all the housework as well as the childcare. (Damn!) It would still be worth my while to work if I were to pay childcare but I thought the calculation worth doing. I don't if know my ramblings are any help but good luck deciding what to do. And you're right, term time only jobs are like gold dust. Actually, they're less valuable than gold dust because the pay is usually crap.