I do understand the position you're asking from OP and I don't think you're being goady. Naive, sadly, but not goady.
The problem is that, when you have children, it's not just about providing basic care for their physical needs and its also about caring for their welfare.
You say you'd imagine that you'd like time to yourself and it's quite possible that that would be the case. But any drive you have to have time to yourself would be overriden with the guilt of knowing that it was at the expense of your children's welfare and happiness.
Even if the absent father isn't abusive and is just a common or garden lazy bastard, you would feel incredibly guilty eg going away for the weekend knowing that your child/ren were sat in the house again in front of the telly, being fed crap food, missing their mum, not being able to go to birthday parties because he couldn't be bothered taking them or not taking them to much loved activities because he couldn't get up in time on a Saturday because he was tired.
Eventually, it would become just easier to do it yourself and you'd have that deep seated contentment at knowing your children - innocent people you'd brought into the world - were ok.
I agree that it should be far less socially, morally, legally (?) acceptable to relinquish your responsibility the way many fathers do. I also think it would be less likely to happen if they played a more active role in their children's lives. Shared parental leave was introduced to, in part, facilitate this I believe.
But, as it stands, there is no legal requirement for a parent to play an active role in their children's lives. So there is no mechanism to 'force' it.