I don't think you're odd, not at all. However, it does sound like your alllergy is making you socially isolated when this may not be necessary.
We all have different ways of dealing with risk. At the moment, I take food everywhere small son - he has a combination of allergies which it make it very difficult to cater for him.
I'm training ds to check ingredients, and not to be afraid to ask about cross contamination. At parties, he always had his own special bag of safe food, so he doesn't miss out, but we also check to see if any of the other food on offer is safe for him. We did take him to pizza expressa of months ago because they have really good allergy information. He was really excited about that.
Ds would totally understand your fear. He is five years old, and cautious without being paranoid, but he had a reaction to some plain chips in a shopping centre cafe recently, and even though he had eaten there without incident dozens of times before, he now totally refuses to go there again.
I'm not saying you should take crazy risks, or move to far outside what you are comfortable with, but it might be worth researching some safe places to eat and sticking your toe in the water a bit. Most people are more clued up about nut allergy than e.g. milk allergy, and you might be pleasantly surprised by how many new options are available to you.