Usually said by mother’s who choose not to work or to work very little. Never heard it said to a man. I work full time. We don’t have a lot of time during the week but we have lovely weekends together. I don’t feel like I’m missing out on their childhood 