There speaks someone who doesn't understand much about mental illness. There can be a major difference between going somewhere you are used to and go to on a frequent basis, and going to a location that is "other".
Things that could impact this difference include whether the off site is enclosed or very open and/or what the journey is like, which could affect the agoraphobia. For some people with autism meeting new or different people can be very difficult, in fact anything different from the usual routine can be very stress inducing.
For reasonable adjustments to ever be taken seriously, there needs to be consideration about the difference between them causing major issues for the organisation and other team members, and the adjustment being mildly irritating to some.
What if the reasonable adjustments are physical and mean no lifting of heavy objects, and occasionally boxes have to be lifted? The rest of the team may not enjoy having to carry a box extra each, but it hasn't impacted them or the organisation seriously. Otherwise you'd be talking about getting rid of an otherwise good worker on the basis of them not being able to do one occasional part of the role that others can easily cover. On the other hand, a disability that means someone can't do the main part of their role e.g. physical disability when they need to be walking about or on their feet all day, may be grounds for considering capability.
All a matter of degree really - many adjustments can impact others even if it's a parent always having to leave bang on time and others having to do any late work in emergencies,.