AIBU to ask the mum if everything is ok at home & mention Rose's behaviour in a gentle & considerate way?
Everyone has jumped straight into invite/not invite and ignored the actual AIBU. No, OP, that wouldn't be unreasonable, it would be kind - it would be you trying to actually understand what's going on and address it rather than make a snap decision one way or another. If something is going on at home, is that something you can talk to your daughter about? Are there ways you and Rose's mum can help the girls work through their issues? Are there strategies Rose's mum uses at home? Etc.
Would anyone here invite a colleague that was horrible to them to their birthday celebration so as no to exclude them if you weren't inviting everyone from work anyway?
No. But if someone I had been close friends with for half my life was going through some shit and behaving a bit erratically, I wouldn't invite all my work friends except them to 'punish' them. And I certainly wouldn't abandon the friendship after less than two months of erratic behaviour.
I would however think of ways to protect myself - not going somewhere with alcohol, asking my other friends to support me if things went a bit tits up, maybe going to the cinema or something where there were limited chances for her to be hurtful.
Ditto this party. I honestly don't know if you should invite this girl or not OP. But I do know that if you DO invite her, your first priority should be to make sure she doesn't have any opportunity to hurt the other kids - whether that's through distraction, distraction, distraction, or having her mum there, or making sure there are enough adults to supervise the kids really hard or whatever. I think teaching empathy is important, but I absolutely agree with other posters that that doesn't mean your daughter and her friends are a punchbag, and I think you can model that by stamping on any inappropriate behaviour at the party HARD.