No I would not go. It would be uncomfortable and extremely tedious for the kids, trapped in a car for hours, with just a break of "trapped in a resturant" in between. And by extension not a lot of fun for you and your DH.
If people want kids there, then the event has to be set up taking into account factors that render it more child/family friendly. Otherwise, unsurpisingly, it may decided that there is too much pain, very little gain, and it gets skipped.
I'd leave your DH to explain the obvious, that 6-7 hours of being in a car/resutrant is too much discomfort and tedium for the kids. If the other adults don't place any importance on that, in contrast with placing major importance on their own disappointment that the children won't be, then they are expressing their priorities. Fair enough. But they shouldn't then get upset if the message of "us first, you and your tinies last considerstion" is heard and understood.
Parental priorities are, and should be, on the kids' needs before adult wants. And regardless of any fall out, that is where the line in the sand has to be. So, no, I don't think it is mean or selfish of you two to give this a miss.