This is so shocking.
He - a married man - arranges to meet up with a woman, and have sex with her. This is while his wife is at a hospital appointment and he's meant to pick the children up from school, but he forgets to do that as he's busy having sex with another woman.
And from your post it seems he brings nothing to the table anyway.
Please stop staying 'for the kids'. You aren't doing them any favours. Your marriage will erode any sense of who you are. It is making you miserable. You will feel resentful and unloved. All these things will affect who you are as a Mum. You can't be the best mum to your children in this situation.
Whereas if you leave you can claim your life and self-respect back. You can create a calm, happy, loving home with your children. They will grow up in a home filled with love.
They are young which makes it easier. It will get harder as they get older and become more aware.
And if you stay, they will see a poor relationship model, which they may take in to their own adult relationships. Much better for them to see their mum be strong and show them you don't put up with poor behaviour from a partner.
You deserve so much better than this man. Your children deserve better than living with a man who puts sex with a stranger over them.