I can empathise with your situation OP, if on a slightly lesser scale. It must be a horrible situation to be in. What I have finally learned over many years is you can’t fundamentally change ppl, or their ideas about you no matter how much you try and want to. You have to accept them for what they are.
But you don’t have to put up with them.
Your mum sounds awful, as does your sister and niece - perhaps this is why they get on so well - they are similar ppl. You aren’t like that, you don’t want to be like that, they won’t change, so the only way you will have a happier life is to cut contact with them. Each time you are interacting with them it is marred by their behaviour or the shadow of their presence when you finally get time alone with your mum. But do you really want to spend time with her, or the idea of how she should be to you?
She doesn’t sound like some sweet, befuddled old woman who’s trying to please everybody and is getting it wrong. She sounds like she is actively trying to play you off against one another so she always has maximum attention and visitors, a triumph to the elderly living alone. I’m sure it’s boosting her ego no end, she was probably the same with other girls at school and boyfriends - thoroughly unpleasant. Did she encourage competitiveness between you as children? Maybe it’s a new thing now she’s old and wants to ensure she is being visited and looked after etc. She does sound like she prefers them, but who knows, she could be acting the same way about you and your kids to them! They could be acting this way through their own insecurities and jealousies etc. who knows.
Bottom line is - you need to stop
dwelling on their dick behaviour and move on.
They are making you feel awful so don’t visit - it sounds like it’s unpleasant for all
parties. You won’t shed your image (real or perceived) as the black sheep, but at least by removing yourself from the situation you won’t be confronted by this torment on a regular basis.
It might make your mother realise what she has lost, or she may dig deeper in with your sister in law etc, but nothing you do will change the dynamic it sounds like, so you may as well look out for yourself and your kids and concentrate on happier times spent with them and ppl who genuinely appreciate and do want to spend time with you.
You will probably get guilt tripped that you never visit - feed them the same lines they feed you - you’re too busy right now, your in laws have invited you somewhere, your DD is taking you out to lunch etc. you’ll let them know when you get back!
It’s interesting that your sister is competing now for your mother’s attention, but left solely to look after her affairs I’m sure she’ll want to know exactly why you aren’t round there as much. You’ll just have to be strong.
A possible outcome would be that your mother leaves the majority of any inheritance to them - but imagine your feelings if you put yourself through this mental and emotional torment and she does that anyway, which seems likely. Perhaps this is part of their thought process - the hopes they can curry favour so she is more generous to them - family members can be fucking awful, but ultimately, a good proportion are always going to be looking out for themselves and their children - there’s not a lot you can do without coming across as money-motivated yourself.
I feel for you OP. At 86 who knows how many years she has left, so I can understand you wanting to spend as much time with her as possible. But I would reiterate the best thing you can probably do is step back as you aren’t making yourself feel any better. Try a therapist and you can always use this as a reason you are keeping your distance. It sounds pretty toxic.
Good luck.