I want to agree with the posters who say that residential care has to be the way forward. In my family we have been through similar very recently, the delusions were not scary for our Mum, but they were making her quite distressed. We were trying to avoid residential care, but Mum lived alone (no family member in the same city and I am not even in the same country). She was not safe. The safety issues were around her declining physical condition and her ability to make decisions about daily tasks.
To cope with the delusions we would sometimes tell her they were dreams. Other times the content of what she was saying meant that we could listen and agree with her. Recently she has fixated on contacting someone who she knew 70 years ago and this has become distressing rather than scary. We try to keep talking about what is happening in the present or watching TV together.
Mum recently needed rehab after a number of falls. I had assumed that NHS staff would be the ones to say that she could not go home. Unfortunately for us they seem to want to follow the patient's preference. Mum does want to go home. I did not believe that if she went home with 4 visits a day she would be safe. We could not use the medical staff as 'Bad cop' for telling her that she could not cope alone. When I would beg her to stay in her chair and wait for the carers to come and get her to the bathroom Mum would say that she was fine, I shouldn't worry. Hmm that doesn't really work does it!!
In fact although in one breath Mum was saying she wanted to go home, when I told her that I did not want her to go home she agreed with me. She said her room was nice and clean, the people took care of her and the food was good. I think her wanting to go home is about familiarity but also in her head she thinks life at home is going to be like it was 5 years ago.
Since NHS staff/social workers seemed to fail to make an assessment based on safety I still had to persuade the other family members that shared care to agree. OP I basically had to say that I wanted to be supportive but I had to opt out of making so frequent visits when there were crises. I did have clear family responsibilities at my home so it came across as being realistic rather than unkind. In your case I would suggest that you have to say to your brother that if he wants to organise a rota you could do a specific number of sessions per month. Make it clear that the rota is up to him though. You have x y z responsibilities that mean that if he wants Mum at home, he needs to do the lion's share. I think spending far too long reading Mumsnet posts has been helpful to me. I started to use the B word - boundaries. It is setting boundaries about what you are willing to do.
To start me off on the road to considering a setting about 3 months ago I did a Google search of care homes in the area. I visited two and was ready to do virtual visits of several more options. I checked out the eye watering prices and make a spreadsheet of what I thought were my priorities/questions to ask. I think this sort of made me feel that I knew what we were looking for.
We have been fortunate in that Mum had a place in a care home for her post-fall rehab. She is just going to stay there. The location is not ideal, but we have taken the decision that taking her to a completely different home would be stressful. I can cope with paying for an Uber or a bus journey of over an hour when I visit 5 or 6 times a year.
I wonder if in your case OP you could do a period of respite as a try out. This was called a 'tactical respite' in one of the homes I visited. Brother would have to agree though. Maybe if he wanted to go on holiday he could suggest that she goes to a setting for a month (they do need time to get used to the setting). The month could stretch until your Mum agrees that the family is better off.
I can hardly recognise myself being so cold hearted, but I told myself that we had devoted two years of trying to follow Mum's wishes. It was no longer an option. I reminded myself that when she was the same age as me she was going on trips, having hobbies and enjoying herself. I can't believe that this is what she would have wanted for me at this point in my life. I have spent a lot of times on planes is the last 12 months. Unfortunately none of those plane journeys have been to anywhere that warrants a postcard home.
I am surprised how I am not feeling guilty with this decision A huge weight has been lifted.