There's (very roughly) around 40 million parents in the UK and around 11 million disabled people.
It doesn't take much to figure out that if parents who do not have a disability, of children who do not have a disability, abuse the accessible facilities for those who DO have a disability... it becomes a problem for those who do have a disablity.
The other irony is that this discussion about whether its ok to abuse the rights fought for by a minority and vulnerable group, is taking place on a forum where a certain group are vehement that a certain group is trying to take over and abuse the rights of a vulnerable group.
The reason accessible toilets EXIST is because disabled people fought for them. The same can be said for parking spaces which is another area where more and more, disabled facilities are being mashed together with parent and child facilities.
Their provision is still insufficient, many do not meet even the basic standards.
On several occasions I have had to leave my wheelchair outside, and go to the toilet with the door open, not into a private area full of other cubicles, but into a shop foyer, into a busy store front floor... into a hotel corridor...
On more than one occasion I have got into the toilet, only to discover that as the toilet seat height doesn't actually meet the requirements, I can't get on it, or if I can.. I can't wipe my own arse.
Once I couldn't get to the toilet without leaning on the sink, the sink was held up by an aluminium beer keg, it fell off the wall and landed on my foot. There was an emergency pull cord but have a guess where exactly it was... tied up so it hung 3ft off the fucking ceiling.
I was once bursting for the loo, rolled into the toilet to discover it was knee deep in piss, because no one had cleaned it. Other peoples piss, all over my shoes, jeans bottoms, all over my wheels and rims and thus, all over my hands.
In one toilet the door opened out into a corridor that prevented the door opening fully, meaning I couldn't get my wheelchair into the toilet, not only did I have to leave it outside but I had the choice of pissing in front of multiple conference attendees, or, shutting the door and leaving my extremely expensive wheelchair unattended, in a building of 1000 strangers.
In hundreds of places, to go to the toilet, I have to go and locate the person with the key and ask permission to go to the toilet... do you have a clue what that feels like as an adult?
I am really lucky, as long as I can get IN the toilet and the toilet seat isn't stupidly high, I am ok. I don't need to change myself, empty a bag, take off calipers or wait until my arsehole stops spewing blood...
There are still people, many of them, for whom suitable accessible toilet facilities STILL do not exist in many places, and their needs mean they simply cannot go out, at all.
So no, it might not inconvenience someone if you just 'nip in' quickly because its more convenient for you... but when EVERY parent with a big buggy does that, then hell yes it's going to be a problem.