I am a father of daughters. When they were young this problem did not arise for me because we lived in a place where public lavatories were differently organised than currently they are in UK.
However, I did have decisions to make more recently when I had the good fortune to be involved with care of granddaughters here. (Moved back to UK, some children did likewise; interesting comparisons with more internationally-based children but I will not expatiate.)
I never would have felt easy going into the Ladies. So I never did. First choice was always a 'baby change' facility, something I had got used to when the children/grandchildren had been in nappies. If no baby change place, next choice was disabled/accessible toilets; in the many, many times I did this I never once felt I had discommoded anyone who needed to use the facility because of disablement. Lucky, I suppose, but, well, my grandchildren came first, anyway.
Of course, even in towns/cities, facilities are not always available. I changed nappies and, later, helped little girls pee, in some, erh, interesting places. If neither baby changing room nor disabled toilet were available, a cubicle in the Gents would serve. And, of course, sometimes when even that was not available, holding a little girl over a patch of grass or even a road grid in emergency situations might be necessary: needs must!
Now all my grandchildren can go alone while I wait outside. Or, more often, I have to say, when we spend time together out-of-doors, they willingly wait outside when I am caught short, which happens more and more often as I grow more and more physically decrepit. (Patches of grass/drains? -- I draw a veil.)
All part of life's richness. But tldr it does seem to me women have the right to veto men going in the Ladies; even if some women think it OK, I know some do not, and that is enough for me (and should be enough for other men too).
[How to deal with a small child needing to pee/a change of nappy, when said child is terrified of the noise of hand-driers? -- That is a whole other can of worms with which I am sure some of you are familiar. ... Like most things in life, it passes, I can report, however stressful it might seem at the time.]