I had an undiagnosed allergy to brazil nuts throughout my childhood & was routinely told I was being silly when I insisted that I couldn't breathe if I ate one.
So I assumed that I just didn't like them, & spent years casually discarding them from mixed nuts (was a veggie student at the time - we ate LOTS of nutroast).
Then in my mid-twenties I accidentally ate a chocolate-coated brazil nut & ended up in A&E, choking & having a massive asthsma attack.
Hazelnuts & walnuts also make my mouth 'burn' - hard to describe, but a maddening burning itch.
Now, I'm obviously NOTHING like as allergic as the OP's ds, & I don't expect anyone (including Dh) not to eat nuts when I'm around.
But if a small child had similar symptoms (ie. not life-threatening but thoroughly unpleasant) to mine, then of course I would remove all known sources of nuts if we were going to be sharing acommodation. If it's a self-catering holiday house then it's no problem at all, surely, to eat a ceral without nuts for a week.
I think milk & egg allergies do require a different approach as they are less well known & in so many processed foods.
It looks as though the OP's ILs simply don't understand the risk, or are choosing not to. I would definitely be choosing to avoid them re: shared holidays in future, & insisting that they acknowledge the very real danger before they had further contact with ds.