I agree with Acekicker - it's okay to be hanging around when your DS is still only 4. You might even find that the other mum appreciates your extra pair of hands.
Dh thinks its important for him to go, i'd rather not send him and save the hassle but i know thats just me being selfish.
No no no, you're not being selfish!! It's completely natural to be worried. What kind of mothers would we be if we didn't ever worry?! When DS was still a baby, I vowed that I would never expose him to the social exclusion he might experience from this kind of party... but I have seen sense and mellowed over time - it's much more socially excluding not to send him.
My DS is now 3 and still allergic to dairy, eggs, nuts, pulses, seeds, garlic, bananas and a few other things. We have just got back from another 3 year old's birthday party. He was amazing! - coped really well with all the food issues and had such a lovely time playing. I'm so glad I took him.
His friend's mum had offered to try and make food that was safe for him, but she was obviously stressed about it (and I didn't fully trust her to do it properly), so I said we would bring special food in his special bag. This worked really well. He clearly understood that he was not to share food with anybody else. There was a wonderful array of child-friendly finger food on the table when we arrived (it was great and all the other children loved it) but DS was quite happy with the sugary junk safe treats we had taken with us.
Even at his young age, DS understands about his allergies, and is very good about being careful (this always impresses me because he is not generally a particularly obedient or tractable child, but if I say, "no, because that food has cow's milk/egg/nuts in it, he never, ever argues!!!)
The party-giving mum had alerted a few of the adults there to DS's allergies, so I felt that I had support and was not the only one watching him (obviously I did watch him like a hawk, but as unobtrusively as possible). At one point, she gave out ice-creams, and that bothered me a bit because they were dripping everywhere (DS reacts on skin contact) but he was so sensible - he stayed out of their way and didn't make a fuss at all (he had Haribo sweets from my handbag instead, and was quite happy with that because they were his 'special' sweets).
In short, go with him, have a great time, and let us know how you get on
