My first has allergies (peanut and milk), too. When my second (now 15 months) was a few weeks old, he started showing signs of being allergic to milk. I trotted off to the the allergist to ask her opinion and came out very happy I hadn't been there before. She is a wonderful doctor, but I have the feeling that she would have had me eat only rice from the time I conceived and then introduced foods under controlled cirmcumstances one at a time. The thing is, the doctors' don't really know what's going on with allergies, what causes them and how babies will react. They are dealing with statistics and are trying to lower your child's chances of having allergies while at the same time not causing harm by denying you or her foods. It seems that everyone has not only a different theory, but also a different idea of what is too "chancy" -- ie, they can see the same set of data and walk away with vastly differing thoughts on what can safely be eaten. It's all in the way a person defines safely.
Here's what I did: saw my doctor when it seemed that the baby was allergic to milk; did a proper food elimination trial to prove that he was; tried milk again a few months later (I tried it, he was breastfeeding) and he still had the allergy; tried milk again a few months later, he was ok and then I gave him a little bit, food challenge style (ie, wipe a little under his lower lip, if no reaction, give him a tiny bit, if no reaction, give him a bit more, etc) and he was fine. The only thing I don't eat are peanuts, but I don't eat peanuts anyway because my elder son is so allergic to them.
My advice is to not worry about eating anything that your elder son doesn't have an anaphylactic reaction to and even if he does, unless it's nuts or peanuts, then just keep an eye out. Actually, I shouldn't call that advice. That's just what I did.
Hope it goes well.