tbh, as unpopular a view as this might be, I think that after a four year affair it is entirely likely that the op is more than just "a meaningless fuck,". After four years an affair will be built on more than just sex, a sex-only relationship just wouldn't be sustainable for that long unless it was agreed to be such iyswim which in this case it doesn't appear that it has been.
It's also possible that the om is no less unhappy in his marriage than the op is, and that like the op he thinks he is doing the right thing by staying, but that actually, leaving his marriage would be the best thing also. We know nothing about him, but there is nothing to suggest he is any more or less of a bastard than the op iyswim. Affairs often come from a more complicated place than just wanting a meaningless shag, especially when emotions become involved and the affair becomes more sustained over time.
But regardless of what unhappiness or perceived unhappiness has led to the affair, none of that justifies the continued deception of partners for four years. Sometimes it's easy to see why someone ends up having an affair, however as it is often having the affair which makes people realise that something is wrong in their own marriages, the answer is not to stay in that marriage and continue the affair in the belief that no-one will get hurt that way.
Op your affair has gone on for so long now that if you did leave your respective partners for each other it simply wouldn’t be a case of finally being able to be together, because the fallout from this could potentially be so huge that it would destroy both your lives as well as those of your partners and children.
For a minute op just imagine yourself in the position of having left your husband and your om having left his wife. Now imagine that:
Your families find your actions despicable and want nothing more to do with you both. As a result your children will never see their grandparents, aunts, uncles, cousins etc again, except perhaps if your ex’s maintain a relationship with your family and therefore maintain the contact between your children and your parents who want nothing more to do with you.
Your ex partners tell your children why mummy and daddy don’t live together any more and that it’s because mummy/daddy wanted to be with instead. In fact they tell your children that om/ow are the reason the marriage broke down and as such your children want nothing to do with you and decide they want to live with your ex’s with eow contact which will diminish as they get older.
Your mutual friends want nothing more to do with you because they have taken your ex’s side.
Then as you live together, every time your new partner speaks to another woman you will wonder whether he has some feelings for her. You will think you see signs of infidelity, a text, a phone call, a comment, all of which could be entirely innocent but because you have helped this man deceive his wife you know what he is capable of. Similarly because he has helped you deceive your husband he knows what you are capable of, and similar levels of mistrust, paranoya, suspicion will apply.
To the outside world you will need to maintain a happy relationship because everyone will be expecting it to fail, and after four years you will need to prove them wrong. After all this is the love of your life right? The man you want to leave your husband and marriage for?
No-one is saying that you should stay in an unhappy marriage fwiw. But for a minute take the om out of the equation and ask yourself if you still want to leave your husband. If the om left you tomorrow would you still leave? And if not, why not?
The om doesn’t hold the key to your happiness, only you do. It is entirely possible that you do in fact love each other genuinely. But if you do love each other, then you need to take a step back from each other to find your own way in regards to where your marriages are meant to go. If your marriages are meant to end regardless of whether the other is in the picture, then end your marriages, and if this relationship is meant to be then it will be at a time when it is appropriate – when you are both free to be together and when the beginning of your genuine relationship is not built on the end of your marriage.