We're wondering about this - DH is veggie, not vegan, but he is also allergic to dairy and I'm lactose intolerant. I eat meat, but we keep a vegetarian home by agreement, which is mostly vegan, and we've agreed it will be our children's choice- their grandparents will feed them meat, and I will on occasion.
I don't know about a dairy allergy but I can 100% guarantee your children will be lactose intolerant if you don't feed them dairy before their teenagers, and as someone who became lactose intolerant through illness I think it's an awful choice to make for someone else.
I actually know three people raised by vegetarians, all eat meat now. No matter what you do or say, they will make their own mind up. However, if you take away their ability to process/digest, they may be v angry with you. I think letting their father cook them meat, or at least dairy, on occasion, and explaining to them why you don't eat it, is the most likely way to influence them anyway. Assuming you aren't planning on raising them to think their father is deeply immoral, they'll know it's a choice. They're much more likely to respond if it's a real one for them.