No I don't think x factor exploits people with learning difficulties specifically. I think x factor exploits everyone. When they start the auditions they are looking for two types of people, those who are very good, and who will likely make for good ratings when they reach the latter stages, people that the public are likely to vote for, and those who are exceptionally bad, and who, quite frankly, make the first few episodes worth watching because most of them are completely deluded and it is hard to understand why anyone who is "that bad" would want to put themselves through that on national television.
One of the reasons why applicants have to be over 16 to participate is because they can't be seen to be exploiting minors, because when you sign up to x-factor you have to sign and agree to a long range of terms and conditions. You don't just show up at audition and go in front of simon and co, you have to complete a long, detailed application form, in which you have to give details about yourself, your influences etc, and you also have to read and agree to a list of terms and conditions, one of which states that your performance is effectively the property of said production commpany and that it can be used, effectively, in any way they see fit. If someone is capable of filling in all that kind of detail, then they are aware that they might be shown making a prat of themselves on national television. And tbh, if someone has severe learning difficulties, then it is unlikely they would be able to fill in all the details on that form without help from someone else, in which case, the person helping them should have more sense than to send someone they care about into a situation where they might be exploited for the entertainment of others.
Imo it's not the job of the production company to decide who has learning difficulties and who doesn't, if they have people close to them who care about them, then those people should be discouraging them from going on the show in the first place.♦