You're focusing very much on schools, work, physical and system environments that are modern though
Humans are social animals, there are social norms, many of which are universal and although they vary to some degree culturally, they're still fairly similar across the species.
Some, if not the majority of people with ASD in particular struggle for example with empathy, social processing, sensory issues and social communication. Rigid black and white thinking is very common.
Those traits would stand someone out and often make them socially and romantically undesirable, a social outcast, disliked at the extreme end.
There are some who no doubt in ancient times, tudor times and Victorian times who will have thrived depending on their role and status in society. People like to throw around that Einstein must have been ASD as was Newton perhaps. But these are outliers, managed to find a way to utilise their focus and obsession etc. The average Bob or Harry trying to navigate through social structures with much more rigid rules for social engagement than now may well have struggled. To be isolated would have meant not eating, not working, not having any way to manage society around you. Modern times conversely have a lot more isolationist and individualist tones which mean some of that is easier for someone who struggles with social integration.