I was about to add an anecdote related to this, but first, I think most (Olympians) are younger and pre children.
I have friends, a husband and wife who each have an Olympic gold medal in their sport. They stayed very firmly involved, and playing age level rep teams for their country now.
When they had children, they continued with their sports, no longer Olympic level, but still beyond the average person by far. She told me that one day, when their eldest child was 7, their child wanted to go to an extra training session for a different sport. She told her daughter that she couldn't go because that's when she (the mum) had training. She went to training that day and informed everyone she was retiring, effective immediately. She said as she told her child she couldn't pursue her own sport that it instantly dawned on her that she'd had her chance, it was now time to support her children in their endeavours.
Her eldest went on to represent her country internationally in a different sport (not Olympic level though). The second child has been to the Olympics in the same sport as mum. I couldn't respect her more for what she did. Instead she has written newpaper articles and done TV commentary for the sport.
Her husband, continued with sport being number 1, they divorced. I'm not saying it's related, I didn't ask that. But he has now lived in at least two other countries, coaching their Olympic team in the original sport.
I have another friend, funnily enough, in the same sport. Has always played in rep teams, but not internationally. She's now in the over 70s team. When her kids were little, they couldn't play sport because it was all about the parents and their sport. They were in their 30s when I met them, so no longer aiming for Olympics. I remember their daughter being very sad when she had to give up dance lessons after one year, as it was too hard for the parents to manage around their sport.
I'm not against sport, but if you're competing at high levels after having kids, then I think you need a 'village' to make sure the kids don't miss out. If you can manage both, go for it, but if your child can't ever do things or really misses out on time with you, then, yes, it's selfish.