I don’t think you are being unreasonable. If you had a set up where you could easily cover child care for your 4 year old, they wouldn’t have a problem with you being away, and you didn’t have work commitments outside of this job it would be one thing, but that isn’t the situation here.
I have to say, though, that I really, really value residential trips and I still have many amazing memories from trips away. Admittedly they were ones run by the local authority rather than my school but the teachers involved went above and beyond and we got so, so much out of them. I am so grateful to the teachers who organised and ran these things, even more so having since been responsible for children on over night stays myself and having become a parent. I still talk about things that happened on those trips from time to time and people are still amazed, in a good way, at how cool the opportunities were.
It was a different time back when I went on trips though. Staff did drink, we drank, staff invited us to their house for an after trip party, there was far less box ticking and the responsibility placed on teachers was more reasonable - they were not expected to be super human. If things went wrong they went wrong, it wasn’t necessarily someone’s fault. None of the above would happen now. In some respects this has improved things but it has also taken away the human aspect. It is now a job which is done as per a contract and not a vocation. I think the staff that were asked to go on some of the trips I went on were genuinely as thrilled to be asked as we kids were to be picked for them.
Ideally you’d look forward to residential trips, albeit in a type 2 fun/ it’s going to be bloody hard work but very rewarding sort of way, but if it’s not your thing, and logistically it would be problematic, I don’t see why you shouldn’t say no.
My child moved school before getting to the right age for residential trips. The new school don’t do residential trips. I am gutted for him, but happy for the staff that they don’t have to take on that responsibility and have clearly defined work hours.