clean using kitchen spray with bleach, leave to soak in, poke the holes with stiff nylon toothbrush or similar prodder.
The concealed fabsoft grows black mould and mildew, the bleach kills it. Once dead it cleans off easier. You will have to do it repeatedly.
Any excess cleaner just gets washed away down the soap drawer next time you run a wash, I don't bother trying to rinse it away.
Leave the drawer slightly open between washes, like the door, so it can dry out which discourages mould growth.