Have you informed the landlord in writing via email flagging it as urgent OP? You need to do this first thing and include photos. Do you rent through an agent, who manages the property, go through agent if they manage? Did you take pics of the mouldy wardrobe?
Give a time frame for a response, say 48 hours, say you suspect this in affecting your baby’s health and needs to be resolved as an emergency,
If you don’t get a response, which should be that the landlord is sending a professional to inspect it within a short timeframe, then contact Environmental Health at your local council raising it as a health issue. You should be able to email them (contact should be on council website) and enclose pics as well.
They need you to have officially complained to landlord first to give him/her a chance to remedy it I think. Environmental Health will inspect and give a notice to resolve the issue if necessary which will be enforced if not carried out.
You are not in a position to worry about being evicted for complaining as you need it resolved or to leave, and if landlord gives you notice, you have proof you have raised it and further down the line, you can potentially make a money claim against the landlord for renting a property unfit for habitation.
The landlord will likely to go after you for the rent for the rest of your tenancy agreement if you leave early and take court action to get it so you can counter that if that happens but you must have evidence from Environmental Health etc.
In the meantime, can you stay with someone for a few weeks until you can see which way the landlord is going to go and get Environmental Health involved if nothing?