No food is completely out - (even honey is extremely unlikely to cause a problem over 6months), the risk is just down to zero by a year so it's not worth the tiny risk. The problem with fibre ise.g. Gov. Advice
"Low-fat, low-calorie and high-fibre
It isn?t advisable to give ?low-fat?, ?low-calorie?
or ?high-fibre? foods to babies"
"Don?t give your baby high-fibre versions
of foods, especially those with added bran.
It stops babies from absorbing important
minerals such as calcium and iron."
Obviously a bit of wholemeal bread isn't going to cause a huge problem, it's more that the fibre may be preventing them filling up on the more nutritious food that they need, and preventing absorbtion of iron/minerals.
I'd say, if it's easy to avoid wholemeal bread, do so, if it's complicated, just go low fibre in other parts of the meal.
(incidentally the phytates - what causes the low absorbtion are a bit of an argument in adults too, about what is actually "healthier", it's not as clear cut as you'd think, before white flour etc. there tended to be a lot more fermented food in the diet which removes phytates.)