From experience.
Im very sorry, but yes you are (unknowingly perhaps) being unreasonable. You are also wrong, your mum can refuse male carers.
Phone the care company and say that your mum only wants female carers. If she is distressed by the prospect of male carers turning up, then tell them that too. They have to take her wishes into account. The care system is fully aware that women born in the 1930s (my mum) and 40s are of a completely different upbringing and mindset than may be expected these days.
Your mum's care company should have a care plan in place, which they should have discussed with your mum (and potentially your mum's advocate, I was my mum's advocate. She would often tell people she wouldn't talk to them, they had to talk to me). Mun's care plan was a ring binder with over 30 pages in it, detailing precisely her visit times, exactly what was to be provided and when, and background information to allow carers to be able to know a bit about her, etc. This preference for same sex carers should be documented in her care plan.
The care company has to agree to your mum's preferences, but there is a "but". In certain circumstances (usually if the rota'd carer is off sick) and no replacement same sex carer is available, then they will have to send a male carer. We agreed on the condition we were warned in advance, and I would go over to "chaperone". If memory serves this happened about 3 times in 2 years.
If your mum has memory problems and confusion, she needs an advocate. Someone to protect her and stand up for her, as well as guide her through this part of her life. This can be an individual, or family as a whole. The carers can deal with her everyday needs. If you haven't already, I would get power of attorney for both health and welfare, and for finance. You will need it, and you need it sorted before she is no longer compos mentis, otherwise you'll need to go to court.
I know this is an awful lot to take on board. I've been there. It will be hard at times. Make sure you have support of your own, and that your family is on the same page.
Good luck and best wishes to you and your family. Take care.