I'm self-employed. It can be very hard to switch off from working all the hours God sends, to generate your own income.
I don't live with my DP so can get on with my self-employment uninterrupted. But when we're together which is 3-4 times weekly, I put that work down. You have to know when to stop, and engage. I used to find that difficult, but made a conscious effort and it's easier now.
In your case it's neither fair nor feasible for your DP to forsake family life for his work. Makes no difference that he's working for self - I think sometimes people think they should get a "pass" for that, just because they're not working for an employer.
No - especially when a partner is just left to do it all, pick up the slack and keep things ticking over nicely, & the income gained from self-employment doesn't match favourably with the volume of hours applied to it. In which case it's a hobby that pays income sometimes, facilitated by someone else working to ensure the bills are paid etc.
If you weren't around, would this self-employment sustain him both in terms of financially, and lifestyle including time with DCs? That's a key issue.
My self-employment does pay, but not enough to maintain life. So I've a part-time job alongside. It's what you do, being sensible about life.
You need a very blunt and direct talk with him, saying all you've said here.
You're right, you will be knackered when you return to work, as his work aka indulging his hobby and making a bit of change on the side from it, and his wishes seem to be more important than family life you and DCs.