OP, YANBU overall. I do laugh at all the women on here coming on to say they did it and it didn't impact family life.... because most of them were out running at 5 am or whatever. In my experience, men do not do this unless very firm discussions are had in advance. So, you can either insist on your own similar amount of time. Or ensure that he understands that HE needs to be the one to sacrifice to make this happen, not you.
When DS was a baby, DH signed up for a marathon. Theoretically, I was fine with this, especially as he'd recently taken on a role as SAHP and I felt it was really really important that he had his own time.
Except... if he was training, he couldn't possibly help at night or get up with the baby because he needed his energy for training. Okay, annoying. But I could live with that. Then, to ensure he was sufficiently rested before training on a Sunday, he'd get up at between 8 and 9am. Then have a slow speed preparation to get going, often only leaving the house at around 10. And of course, the runs were long.
One day, having gone for a 3 hour run that took 4 because he got lost and having left later than normal because he'd overslept and faffed about, he came in at 2pm. I was sitting on the floor with DS - who had been up since 5 am and had not napped. I wasn't even angry by that point as I pointed out that on mothers day I'd been alone with the baby since 5am.
It was the shock DH needed and he still apologises about it nearly 10 years later. After that, he was up and out of the house by 7:30 on the weekend and if that meant going to bed earlier than he liked or going for a run while tired, then so be it. This self indulgent crap where I was left literally holding the baby disappeared and I was happy to continue to support him as he ran a couple of other marathons over the years.