No, YANBU. There are two issues here, though.
Working from home is tough on its own. There are some people who just can't seem to comprehend that it is real work. My PILs were terrible at it when I started - they'd always interrupt me, bring me random children to "babysit", come and talk to me, ask me to go and get things from town, send huge deliveries to my house...Now that FIL works from home too, they leave me to it, but they just couldn't see it the same way as when I go to the office. You have to be really firm, and eventually it gets through.
Your DH sounds like a knob, though, and that will be the bigger problem. For a start, he doesn't tell you to get a job. He can talk to you about needing to increase your income, if necessary, but then you need to look at all options - including working a normal 9 - 5, if it would earn you more, and covering childcare. If you do choose to work outside of normal hours, so that you don't need to pay for childcare, your DH should be expecting and willing to look after the children. They are part of his family, and he cannot sponge them off onto your parents. Don't allow that to be an option. If he won't have them at the weekend, find the costs of out-of-hours childcare and present that. It'll almost certainly make any increase in wages irrelevant.
I couldn't afford to only earn £600 per month, so he's not unreasonable to mention it if you're in the same position. You have children and the costs that come with childcare and sickness to think about, though, and he has no right to demand that you do anything - especially working all hours, and not seeing your children. You have to find a workable compromise.