Okay, first of all you HAVE TO KILL THEM otherwise you will not eradicate the problem. The best thing to have is rentokill poison traps - they have mouse food inside, the mouse crawls in, they eat it, they run off, they die, quickly.
Either that or kill traps but smart mice can outwit them. If you are to use kill traps use peanut butter/meat as bait. Something sticky (chocolate works too) but it's important they can't just leg it up, nick the food and run off without setting off the trap - nothing more irksome than having a mouse sitting there scoffing your minstrels in front of you on the mousetrap then ambling off
In terms of cleaning up afterwards, if you have seen them, you've most likely had them since you turned your heating on this year - sorry. Your baby will have some immunity anyway, but vacuuming is a good start. I dilute 1 part bleach and one part washing up liquid to twenty parts water and wipe all surfaces. ESPECIALLY if you have breadboards etc.
NB I found a dead mouse in my daughter's toybox and so everything in it was put in the washing machine or dishwasher! Teething toys can normally be put in the dishwasher. But obv. if you don't think the mouse has been near there don't fret.
You also need to find out where they are coming in? And living/nesting. The Rentokil traps are best because they tend to kill the mums and then their babies die (yep, harsh I know).
Under kitchen cabinets is a favourite spot. If you can get a biro in there, a mouse can get in. Block access if you can find it.
Above all don't get too stressed, the little blighters have probably been there months without your child getting ill.