Yes, I think people really do like to stretch definitions to try to justify their own beliefs. My DN is 9 and he'd be horrified if anybody referred to him now as a 'little boy'. I've even heard people refer to women in their early 20s as 'young girls' before - although, oddly enough, never men of the same age as 'young boys'.
How is a child supposed to actually grow, develop and eventually learn to be independent if he's told "You're a big boy now, so you use the boys' toilets all on your own now you're at school" and then, several years later, "You need to come into the ladies' toilets with mummy"?
What is he actually to think? On a personal level, he will get confused about something that really should be straightforward; and he will also learn that you just pick whichever toilet you prefer each time - according to YOUR preference and nobody else gets a say or is allowed to object. As he reaches puberty and is starting to be curious about female bodies and have sexual thoughts, he knows that he was fine to use the women's toilets when he was a couple of years younger, so what's really stopping him now?
And I know that we're talking about toilets here - where most users will generally be behind locked cubicles when anything intimate is on show, so it's 'only' their single-sex privacy and dignity at stake and not a particular risk of being sexually objectified and having your intimate parts seen...
However, if you make these socially-accepted 'rules' that older school-age boys can freely use women's toilets, it's only a logical sidestep to changing rooms as well. After all, if men minding their own business using a urinal, with only their clothed backs visible, are assumed to be a threat to boys, how can you not extrapolate the same to a changing room, where men will quite legitimately be visible partially or fully naked, in the showers etc.?
Such a blurry lack of boundaries is not good for the boys themselves, it's not good for women and girls, and it's not good for society in general.