For me the harmful stuff-to-avoid-in-pregnancy falls in two categories.
The first is things which are probably Ok in small quantities, as it's excess that might cause harm: alcohol, caffeine, vitamin A.
The second is things that actually are generally ok, but if they go wrong they go wrong completely and spectacularly and it would only take a one-off going wrong to cause harm. So I mean listeria risk from pate, salmonella from dodgy eggs, falling off a horse.
The problem with the first group is that it can be impossible to know how much is too much. With the second it's that it's impossible to predict.
IMO for the sake of 9 months (well, actually 8 as one doesn't know for the first month), and the risk of long term feeling guilty if something happened and you were not sure if you had contributed, I'm just avoiding the lot. Een though I did not in my previous pregnancies. I know that I am still horse riding, but now that I am 13 weeks, and my uterus is no longer fully within my pelvis and therefore not as well protected I am going to stop. Not worth the stress. And avoiding these things is relatively easy.