Only 8% of single parents are men. They are unusual. There are many reasons for this, some cultural (childcare being a female cultural norm, the dearth of paternity rights, etc) and some biological (the inability of men to gestate, labour and lactate means that women get their foot in the door first when it comes to establishing parenting roles).
As a result of this, the automatic assumption is that the woman is the primary caregiver. The one who kisses bruised knees, reads stories and tucks a child in bed every night. This is the reason women are treated with much more disdain when they walk away from their children. Everyone wonders how you could abandon a child with whom you've established that bond. We still tend to assume that men aren't that involved - particularly with younger children - so we are more matter of fact about it because it doesn't seem to be quite as big a betrayal.
It's all based on stereotype, but like a lot of stereotypes it is based on some truth - by far the majority of young children are cared for by women.
When women do walk away from children MH issues (including PND) seem to feature more significantly than average, so it's a logical response to ask what is going on at home rather than just assume a motive of pure selfishness.
I would always want to know the circumstances behind someone abandoning their family before judging because life is a complex thing that is often very different to outward appearances. However, in cases where parents of either gender abandon children for no reason other than to pursue their own selfish agenda, I judge harshly. Children come with responsibilities, and while I don't advocate self-sacrifice, a certain duty is to be expected. Abandonment is one thing, however. I have no interest in whether a mother or a father leaves as long as regular, positive contact is maintained, and I would assume that the parent left with the children was the primary caregiver before the split.
All that said, I think it's a fallacy to think that abandoning children is abnormal human behaviour. Worldwide and throughout history it is depressingly common. 1 in 5 children grow up losing contact with their father. In addition to that, right now, here in the UK, 1 child a week will be murdered, More than 50,000 are on child protection registers, 1 in 4 children will be abused, and 1 in 20 sexually assaulted. Mistreatment of children by their parents, including abandonment, is very, very common by both genders.