I think SaraNade is making a point forcefully, but just because it's loud, I'm afraid that doesn't mean it's not right. I think she makes a crucial point in this discussion of how the OW will be seen and accepted.
The moral assumption of 'you deserve to be happy so go for it' is, ultimately, that your happiness is more important than someone else's. That your happiness is so important that hurting others is reasonable collateral damage. This is the bit that some people will find too much, and the cost of the relationship too high. Others will feel it's ok, and go ahead.
And some will feel bad, but judge that the original relationship, the marriage, is doomed or finished anyhow, so it's ok to salvage a partner from its ruins. In this case, the other partner in that marriage should also be in the same place and therefore not mind handing over. Which is often not the case. Hence the upset and sense of transgression.
That's why I say how the OW behaved is crucial in future acceptance. Because by having been part of the affair and likely deception, she's already seen as morally bankrupt, and an aggressor who has hurt loved ones, including children. Hats hard to come back from, and tbh you probably have to be v decent and v selfless for several years, as long as it takes.
If, as does quite often happen, the guy is eventually unfaithful to the OW as well, that's the point at which the family may be more accepting. As then they feel the pitch has been levelled.
The cost of putting yourself first is always going to be high. If you run someone over to get into the Whistles sale first, you can't expect them to be nice to you just because you're sorry that getting what you wanted meant trampling them. That's the deal here.
My advice would be: keep a very low profile. And hope over time they decide to give you a chance. But don't feel they're obliged to.