I think it is because there are soooo many threads where the ultra marathon / marathon runner / triathlete (or equivalent) disappears for long periods of time when they have a young family, leaving the other parent to pick up the slack.
Also things like - really fussy diets, all the extra washing, all the extra kit etc.
We have a shed that I couldnt put a single thing in because it was FULL of all my husband’s bikes. It was a nightmare trying to get my DDs’ bikes out because he was just so thoughtless and inconsiderate with how he stored all his kit. This is just an example and a personal and specific frustration.
Another reason is that these hobbies sometimes make people so self absorbed ie endless talk about whatever target it is with NO interest in what anyone else is achieving or working on in their lives.
I recall ENDLESS monologues about personal bests, Strava, routes, bits of kit, etc etc. Bearing in mind that I don't do this activity because I am not interested in it, it gets very wearing very quickly when that is the only conversation.
I am not saying this is you - you sound like you have a balanced set up.
I am just explaining why some people (me?) might have a visceral reaction to these sort of sports.
Good luck for your ultramarathon.