Don't settle for this OP.
Make some changes, whichever suits you best.
See a counsellor and talk it all over with the counsellor. The counsellor might have some helpful suggestions for you. If not, you can at least vent your anger and frustration.
Talk to your DH (sounds like you're tried this already though). Tell him you're not happy with his constant vanishing acts and he needs to step up more with the DC. You both agreed to have your DC (well, I assume you did) so you both need to step up and parent them. Also, if he's due to finish work at 5, then he finishes at 5 sharp.
Tell your DH you want to see a marriage counsellor. It might shake him out of his complacency. And he might be more likely to listen if there's an independent, third party pointing things out to him. You're heading for a divorce anyway if the situation continues as is. That resentment is just going to build and build until you can't stand the sight of him anymore. And your DC are going to pick up on that resentment, irrespective of whether you stay or you leave.
Draw up a schedule. He's responsible for the DC Monday, Wednesday and Friday nights. You're responsible for them on Tuesday and Thursday nights. Swap over the following week. Or draw up a list of tasks - you will do a, b and c - he will do d, e and f.
Sign up for evening classes or arrange to meet a friend/go shopping on a Saturday. If you're not physically there, then you can't be the default parent.
Why did you cancel your exercise classes? You should have read him the riot act the first time he made you miss your class. Or told him you were dropping the kids off at his office. Then he wouldn't have made that mistake again. Instead you played the martyr and cancelled your class.
I'm sorry if I'm coming across as unsympathetic. I'm not. I have huge sympathy for your situation. Working and parenting small children is really tough and it must be so much harder when your DH isn't doing his fair share.