Hi OP I had to reply to you at length because I feel for you so much. I have been in a very similar situation.
My mum had profound mental heath problems from a young child. Depression, anxiety, a couple of episodes approximating psychosis when a teen. She also had a terrible hard life - tragic bereavements, difficult relationship with her mum, abusive marriages. When she was approaching 60 it came to a head when her then husband (who she already had a difficult relationship with) had a stroke and then got cancer, she was thrust into a carer role she was totally unable to fulfil, there was mutual abuse, it was awful. Eventually they broke up acrimoniously and he moved out. She didn't cope well with being alone, leaned heavily on booze to cope (always a crutch but then out of control), had money worries. She became paranoid that her ex was poisoning all their friends against her.
I have always been enmeshed with her, but the timing of this crisis was difficult as I had recently had my first child and it caused me huge emotional upheaval as I reviewed my childhood and our parent-child relationship in the light of my own experience of motherhood. So I was dealing with some hurt, resentment and bitterness that was very new to me, whilst also trying to love and support her through this crisis from a long distance away with a small child. It all felt too much.
Then she made the suggestion that she wanted to sell her house and move in with us.
I said no. I felt like I had to, for the good of my child and myself and my relationship. I didn't want her MH and her drinking around my DD long term. So I said no.
She was initially furious and cut all contact with me, but did in the end get back in touch and we resumed our usual relationship - so I thought.
I later found this was around the time she resolved to end her life, which she ultimately did some few months later, the day she moved the last of her ex's things out of her house, and the day her mental health nurse missed a home visit with her without advising her it was cancelled. When I read her messages to friends etc (she specifically didn't leave me a message) she spoke about having been abandoned and let down by everyone, friends, family, everyone.
There are no two ways about it, to some extent the decision I made to say no contributed to her death. And it has taken me a lot of time to learn to live with that. In many ways I am still trying to come to terms with it, and forgive myself.
But - and this is the crux of the matter - I would still make the same choice, if it came down to it.
I would never have been able to 'make her happy' or 'make her better'. These were deep profound problems that had existed long before I did, that had endured a lifetime of me trying to fix them, that were worsening all the time. Having her live with me would have broken down the only boundary i had with her, that of distance - all my other boundaries, all the normal boundaries of a parent and a child had been overridden long before I had any choice in the matter. I would have been pulled under with her, I could not pull her out.
And most of all, I was right to prioritise my child, to put her wellbeing before my mother's. If it had been put to me as a straight choice, even then, to lose my mother or to expose my baby at close quarters to all that dysfunction in her formative years with God knows what effect - well to be fair, I know personally some of the effect - well I don't know. I don't know if I could have made the choice. But if I look at it cold-eyed after the fact, try to ignore the huge love and responsibility I felt and still feel for my mum - the choice I unwittingly made would be the right one. Because my first responsibility is to my children. Has to be.
So that's my story. That's the 'worst case scenario' of you saying no to her, and someone who has been through it telling you, for what it's worth, that I would not judge you for choosing your children. You sound like you're doing and have done so much already for her, stuff a child should never have had to or experience. You have given up your own boundaries, and may not be able to redraw them again as the lines have long since faded away. But you can and you should draw an iron boundary round your kids. You decide where that line is but don't for a second hesitate to draw it and defend it. It's your main job as a mother, and the only way we can avoid rewriting our own histories.