Agree that for this trip to be sanctioned, there will have been a risk assessment, so the risk of breaking equipment will have been covered, and guidance given as to what to discourage (eg swinging REALLY high). Afaik, the 1-14 restriction is, more often than not, to give police/wardens/whoever an excuse to kick out older teenagers using it as somewhere to sit and drink supermarket cider.
And if properly supervised, I think there is no reason why this group should not be able to use the equipment - providing that is what they want to do. Learning Disabled is an incredibly broad term, and some adults with LD really don't need or want this sort of activity, and others will keep their delight in it well into old age. But it does sound like they were having heaps of fun.
And tbh, I don't think there's a huge difference between saying that your child is going to be scared of the "weird" noses that people with learning disabilities make and saying that Cerrie should be sacked from C-Beebies because having one arm might frighten the children. They're usually only as frightened as we teach them to be. Sneaking up behind someone and screeching in their ear is another thing, but if you have a screecher on your hands you know about it and plan accordingly.*
However - sounds like a particularly cr@p set of carers. So no, YANBU to be concerned about kids sharing a play space with adults careering around dangerously with insufficient supervision. The risk assessment should have made it quite clear what level of supervision was needed, and I know with very high-supervision groups (ie 1 service user to 2 carers) they tend to split up and go in smaller numbers. The carers chose to ignore the rules because they would rather have a natter and turn a blind eye. The sad thing is that if someone does get hurt (and tbh quite likely to be an adult not a child: the flip side of being heavier is they hit the deck incredibly hard where a kid would just bounce and probably cry but be OK) then it will be no more trips to the swings for a group who possibly looks forward to it all week.
*Unless you have been severely pissed off by someone passing judgemental comments in the supermarket. In which case you might just wheel your screecher up behind them when they are rummaging in the veg aisle. And wait with some satisfaction to see how high they jump. 