The other thing to consider, if the morality of being complicit in the betrayal of his partner and family is not enough for you, is the impact in your personal and professional reputations if/when the affair is revealed.
Let’s face it, if this gets out you are going to be viewed as the other woman who screwed over a woman caring for an unwell and profoundly disabled child. He is going to be the man who betrayed his partner whilst she cares for their sick child, having given up her career (and probably hobbies/social life etc) to do so, I doubt that revelation would go smoothly for either of you. Of course he is more culpable as it is him who is betraying his partner and family, but you are totally complicit in that betrayal and it’s utterly reprehensible. I think you should both be ashamed of what you have done.
I also think you are either utterly naive, or practicing a great deal of self-delusion though. If he wanted to leave, he could and would. He would be absolutely be justified in ending his relationship if he does not want to be with his partner- he can leave and still appropriately financially and physically support his children, so he doesn’t have to stay. After all, he manages to find time to spend with you despite it all, so he clearly doesn’t invest all his time in his family anyway. At best he is too cowardly to admit he wants out for fear of being negatively judged and/or because of the financial implications- and that won’t change. At worst, he’s happy to betray his partner and family and spend time and energy he should be using to support his partner and children on you because he gets something he wants (sex, excitement, thrill of sneaking around, an emotional outlet that doesn’t have the same responsibilities and stresses that he has at home with his partner).
Rightly or wrongly, you will be viewed just as, or possibly more, harshly than he is. Some/most of your friends and acquaintances will disapprove- even if they don’t say it. I’d be so disappointed in a friend who knowingly did this, even if I thought they were an otherwise reasonable person and it would make me question if I had made the correct judgement about their character.
I imagine some of your work colleagues would be shocked too. but it might impact professionally because you are seen to be untrustworthy or showing poor judgement. I’m not saying you would be “officially reprimanded” or anything like that, but there could be a “softer” fall out which has the potential to be just as damaging professionally if you are viewed as having poor judgement, not behaving morally etc. it’s not inevitable, but it is possible. So if the moral argument does not sway you, perhaps the prospect of social and professional stigma might.