OP, your response is perfectly rational. There is a lot of data on this, all easily accessible from government and Ministry of Justice sources and covering England and Wales.
96% of sex offenders whose crimes are discovered, reported, make it to court and are found guilty are male. Huge numbers of men commit sexual offences that are never discovered, reported or fail to reach court. Sexual offences are on the rise. This is the scenario not just in the UK but all over the world.
Given the fact that almost all sex offenders are male, we have formally and informally created a safeguarding system to try to protect women and children (who are the vast majority of victims). It's the main historical why nurseries have traditionally been staffed almost entirely by women (there are others including the low pay and the low status attached to such work). It's the reason why we have separate male and female loos and changing rooms. Because it's safer for everyone. Decent men understand this and wouldn't want to run the risk of being accused of predatory behaviour by going into women's single-sex space. The men who want to get into women's single-sex spaces are the ones we need to worry about. And given that most nurseries are run entirely by women, this man stands out.
Male sexual predators often plan their lives around opportunities to abuse. Trainee social workers are asked to look around their class to try and identify the abuser(s) among them — because professions like social work attract people looking for opportunities to abuse. Think of all the Scout leaders, sports coaches, church leaders, children's home workers and so on who have been found guilty of abuse. Think Jimmy Savile and his hospital visits to vulnerable children...
So, OP, every concern you're feeling about this man at the nursery is justified by reality. You are finely tuned to protecting your little girl. Your gut is telling you that this guy is a potential danger. And you're right, he is statistically more dangerous when it comes to sexual assault than the women in the nursery. That doesn't mean he's actually a sexual predator or that he will assault your child, but the fact is that he's far more likely to do so than a woman. It's nothing personal about him as an individual, except for his sex.
All the women here prioritising equality over women and children's safety are trashing safeguarding. In an age when child porn, cameras and the internet are spreading like wildfire: when ordinary 'nice' men are watching extreme porn on a nightly basis: and when women are falling over themselves to be kind to dodgy blokes (drag queen story time anyone?) women have to be even more careful then ever. Unfortunately. No one is sadder about that than me.
If I were you, OP, I'd ask the nursery lots of questions about safeguarding, about staff not being allowed to take children off privately and so on. I'd make it very clear, and keep repeating, that this is about the statistical risk, not about the individual concerned. It's not pleasant for him: it's not his fault that people of his sex commit almost all the sexual offences. If he's a really decent man, he'll understand the concerns and won't take it personally. But none of us should be putting our kids at risk because we want to be nice.
Sorry for the length of this. It's a serious issue and it deserved a serious response.