I don't disagree with PP that damp is a concern, but just to put a slightly different spin on things....
Assessing for a mortgage is about accepting a known risk. If the survey report isn't specific about the cause of the damp, your bank won't be able to offer a mortgage as they won't be able to assess the true risk.
Has your bank refused to offer a mortgage because the damp is so utterly terrible? Or have they simply said that they need to know more about the damp, including the extent and the cause? To me, it sounds like the latter.
If they thought the house was unmortgageable they wouldn't be recommending a damp specialist.
For context, I was an underwriting manager - not in mortgages, but the concept of assessing risk is the same. If the bank doesn't know specifically what risk it's being presented with, it can't offer terms. So the fact it's refused is simply because it needs more information. As do you.
Damp is the thing we're all scared of finding on a survey and you do need to be very careful or else you'll end up with a money pit.
However, we had some damp show on a survey and it turned out to be because the gravel outside had been built up too high so it was above the damp course. It was a bit of a fixer-upper and they had no working boiler, a man (who used to travel the world for his work so was barely home) and his uni-student adult son were living there and they used to dry all their clothing inside (in the non-heated house) so that was making it damp and unpleasant too. We got in, put in a new boiler etc, and we also knocked down the damp wall as we were extending - but the surveyor said that sorting the gravel would have resolved the damp issue if we hadn't have done that. A year on, and it's absolutely fine. No mould, no damp, and no issues with construction.
All I'm saying is that it's not always disastrous. So yes absolutely proceed with caution but don't write it off until you get proper information. It's a surveyor's job to be cautious and to warn of the worst - but the word "damp" can cover a multitude of sins, and some are more concerning than others.
If you can tell us more about what the survey said we could be more specific.