Hi Op.
I have had a similar situation. I am physically disabled following an accident, and my DH struggled to understand the implications.
What helped for me:
1 writing a sentence that I then used repeatedly if he asked me to go out somewhere in the car. Mine was - I find that very painful, I'd be happy to try to find ways to make it less painful, but at the moment it doesn't work for me.
This allows open the possibility of changes that might make the activity possible, while making clear you aren't prepared to do the activity with out changes
2 brainstorming some possible changes that might help, for me the problem was my foot, so I said we can try getting a cushion for my foot, we can try a special support boot, we can try putting custom foam in the footwell etc. I said I was prepared to trial alternatives on a short drive where we both monitored pain.
3 my DH wasn't keen to make any of those changes so I encouraged him to go with other people/go solo.
4 over the course of about 4 years he did repeatedly ask me to go with him, but I just used the same sentence each time, and then said I was happy to trial mitigation measures.
5 it was about 4 years before he got bored enough of going on his own/ran out of other people to inflict it on, and then he was prepared to trial various foot rests etc. We're now in a position where it isn't too painful for me and I'm prepared to do short trips, but some of the time I still say no.
Please feel free to pm me for advice, chronic pain is not simple, and it can be a complex process trying to work out a way to minimise pain so you can still be involved in a family activity (if you want to be).