Speaking as someone who's attended and led many primary school residentials in my time, I don't think YABU.
As previous posters have said, although I loved sharing some of the experiences with my classes it is incredibly hard work. Many was the trip where we took snacks to eat after the kids were in bed and never touched them all week! Sleeping with one ear open, constantly head-counting, sitting on the draughty floor outside kids' bedrooms waiting for them to go to sleep so you can finally go to bed, dealing with homesickness, travel sickness, vomit, accidents, bed-wetting, meltdowns, fall-outs between friends and just being responsible for the safety of 30+ of other people's children while having to leave your own children to manage without you for a week. Plus when we got back to school parents would complain if we were delayed by traffic, hardly anyone said thank you and on more than one occasion I waited for up to an hour for a late parent to collect their child so that I could finally go home to mine, have a hot bath and a proper sleep.
Then there was the year we were away with a group of London children when the London tube bombings happened and a stupid woman came running up to us and (in front of children whose families were in London) said, "Oh, is your school from London? It's just been on TV that bombs are going off there and loads of people have been killed!"
Yeah, the aftermath of that comment was fun for us school staff to deal with...
OP, I've known more than one occasion when for some reason a class teacher/TA hasn't been able to go on a residential (e.g. because of pregnancy) and what's always happened is that another staff member has swapped with them and the teacher who didn't go has covered the other person's class at school. I've done it myself- taken someone else's class away. So with enough notice to your HT I'm sure a swap could be arranged.