I too am finding it hard to work this out. In one breath you say that your sister has done some bad things, but has matured and is now nice, you enjoy spending time with her, your BIL and their children. She is lovely to be around, your partner likes her, her husband is lovely and you like her children. In the next she’s a creepy, mean-spirited, racist bully. Which is it?!
I strongly suspect there are a lot of difficult family dynamics here. Few people act the way she did as a young person if they are happy, well-adjusted people (there are a few, but most people who act like this have problems of one kind or another). The way you describe your mother, I do wonder if there has been a damaging relationship there. Remember your experiences of your childhood may not match hers… and the truth might be something in the middle of all of the siblings experiences.
I also wonder if your brothers are quite as saintly as you believe. It’s easy to fall into a narrative of one person being 100% in the wrong and every one else perfectly right/been the wronged party. I’m not saying your sister has not done things wrong- it sounds like she has- but to have all the wrong-doing only on one side is unusual.
Of course, none of that excuses her bad behaviour in the past, but it might explain why she did certain things, and might be a starting point for trying to move on. Nor would it excuse racist comments or mocking a child. If that happened (and you are sure)- it’s fair enough to express your absolute disapproval and that if she doesn’t change/apologise then decide you aren’t going to associate with her until that changes.
As for her having to start mediation in order to earn a place at your wedding? That’s deplorable. Mediation should not be used as a stick to beat someone with. It only works if all parties are willing to work on the issue. If they aren’t it just a cruel way of trying to get someone to grovel for forgiveness that you have no intention of granting. If that is how your family works, it might explain a lot.
Ultimately, if you feel you cannot have your mother/brothers and your sister at your wedding (can you not ask them all to be civil just for one day if you really do want your sister there?), then you need to chose who you want there more. From what you’ve said it is your brothers- and that’s ok if that is how you feel.
But you then also accept the consequences of that decision- your sister may well be devastated, as you fear she will, and it could possibly damage your relationship…. But if she really is this racist, creepy, bullying horror you intermittently depict then I’m not sure that you should be all that bothered if your relationship becomes more distant…
…. which makes me wonder if you know that things are not quite as straightforward as your sister is a horrible person and the rest of your family utterly innocent.