How does this limit you enough to be bad for the baby?
You can still eat a full range of fruit and veg, meat, fish, pulses, pasta, bread, cereal, pasteurised cheese, rice, etc, etc.
More people have problems because they eat processed shite than because they don't eat a bit of Brie or some liver.