He's trying to control you with his emails by badgering you at unreasonable times. Block it and contact through a third party - be it a solicitor or a relative. I also think the argument about finance and custody are linked: more custody means less maintanance. In reality, someone who wants to leave a 4 year old at home for a shag doesn't really want the responsibility of lots of contact. He's just arguing so that financially he isn't hit as hard.
Stick to your guns and go for what you are willing to concede. Disucss the behaviour at bedtime and considering leaving the 4 year old alone, your son's anxiety and also the ow's alcoholism. They all play a part in custody arrangements, though he will probably argue that your illness means you are less capable to care for them. So you need as much evidence as possible to push for what your dc need, which is stability and to see dad regularly but for short periods of happy, quality time.
Yes, you'll have the burden of the school runs and the day to day running of things, but just think that when he has them e o weekend you can then go out for meals with friends, cinema, stay in and binge watch netflix etc whilst he struggles playing happy families with someone who must be a bit of an arsehat to have had an affair with a married man like that.
They'll have their problems. They're both cheats and selfish. So there will be conflict.
The children will cope with some dad time, even if he is shouty, as they will then come back to you and at least they don't have him shouting at them every night like he used to. It will be an improvement for them.
I can't do any liasing with my ex and it all goes through a third party. He is just not able to give a shit sadly.