DP (Builder) says you should don a good quality dust mask (Spores are sometimes irritant/poison) and physically remove the fungus by scraping and then scrubbing with a stiff brush.
Once you're sure you've removed it all you should treat the area with an anti-fungal wash which you can buy at the builders merchants (Generally DIY stores will only offer weaker solutions). Make sure you read and follow the label as these products are also irritants/poison.
He thinks it'd be a good idea if you investigated the source of the damp as there's a possibility you could end up with structural damage if left unchecked.
Start by checking for blocked/broken rainwater downpipes and drains. Blocked air bricks. Ground levels which are too high. New paths which are directing water towards the building rather than away. If there are no obvious causes it may just be ground water which will need more drastic (and unfortunately expensive) remedial work.