I didn't eat wheat, yeast or dairy products for two years due to a skin complaint. I had tried all sorts of things from the doctor that didn't work so had a food intolerance test done by an alternative health clinic.
I felt great on it. Skin cleared up, never had a cold or any kind of bug for that time and generally was in tip top shape (as well as being digustingly skinny - which was a novelty!)
Started eating all that stuff again when I was pregnant as I felt that the baby needed calcium and I needed the carbohydrate and miraculously my skin did not flare up. Even after ds was born my skin has remained clear!
Do suffer from colds now and again but not as much as before and am not model like slim anymore but blame that on ds!
I still do avoid dairy - am a firm believer that I suffer more from catarrh if I drink milky things. Only drink black tea and coffee and have soya milk on my cereal.
I wish I could go back to wheat free but it is very difficult to stick to considering the fact that most things have wheat flour in them (just take a look at the ingredients on the back of packets.) I try to keep bread to a minimum but don't mind pasta. I think yeast is the real baddie!
If you do go down that road, Waitrose do a really nice yeast and wheat free rye bread. It has got a russian name. Most other wheat and yeast free bread are pretty yuk.
More and more stuff is on the market to help diets like this. Do a bit of research before you start though!