It's ridiculous. The main reasons mothers take infant boys into female toilets are
a) because they still need help with aspects of toileting and hygiene
b) safeguarding from adult men (though admittedly this cut off is a grey area because boys over 7 are still vulnerable so IME it's a risk that has to be managed carefully by their adult carers for a few years)
If RMW is suggesting there are adult men who fall under a) then clearly we are talking about disabled men who will have at least one carer with them. In that situation they are catered for with accessible toilets for disabled people.
Point b) is a non starter unless you are arguing that every type of man with a vulnerability should be able to use the female facilities - old, gay, teenage etc.
If anyone does want to argue this then they need to explain how if adult men are not safe with other adult men this magically makes it safe for women and girls (arguably more vulnerable than any man) to share spaces with men.