OK, in a way it is a positive that you are pushing back at those who are questioning whether you are being a good mum.
I get what you are saying. You haven't ever allowed him access to your children, so they are at least protected from DIRECT harm.
What they are not protected from is the indirect harm that comes from the harm that he is causing you. The effects on your mental health of being with him are bound to have an impact on your children. The very real risk that he could cause you physical harm is there, and would also have a massive effect on your children.
Now, there have been many comments from other women. This is coming from a bloke (so no chance of any "all men are bastards" bias here).
He has taken you to a point where you are prepared to accept clearly unacceptable behaviours. He pushes the boundaries, until he judges a line that you won't cross, then he steps back just the other side of the line, and you praise him for being a better person, oblivious to the fact that he isn't a better person. He is a shitty person, and being marginally less shitty than last week doesn't matter.
He is clearly a narcisist, he will never change. Sadly, when you get rid, he will find sonebody else to feed off, and the cycle continues. Even when you get rid, he will have a plan that when his next victim wises up, he can have another try at feeding off you.
I know that you want answers as to "why", I know that you want to tell him some home truths, but for as long as he has access to you, he can twist things, and make himself the victim here.
Deny him that right now. Big girl pants on, and accept that you will never know why he is like that (and even if you talked for 20 years you wouldn't get to the bottom of that). Accept that you don't get to tell him straight (he wouldn't actually listen anyway, and denying him communication will actually tell him straight FAR more effectively.
Please, stop saying you WILL block him, and do it right now. Do it before he escalates even further, and your children end up without their mum.