It's not the asking, it's the unreasonable amount of time, the lack of a sense of fairness, the refusal to take his share of family responsibility, the petulance and unreasonableness of throwing it back at you, the sheer selfishness.
Only you know what the whole picture is, but from this little bit of the picture, I'd say this guy either doesn't know what fatherhood entails or knows and isn't prepared to do his fair share, and certainly does not see you as an equal partner/equal human being. And I would be having a serious think about what that means for you and the family.
Also, I can see that there's a "type" who gets into cycling this way - but I don't think it's the cycling per se.
For example, my brother is a big cyclist, lost count of his bikes, calves like gnarled tree trunks, Strava star, he also works away from home then gets long stretches of leave. Does he use massive chunks of his leave to fck off on self-indulgent 3-week cycling trips? Does he fck. He spends his evenings while he's working planning ridiculous skiing/ travelling/ camping /canoeing/cycle touring adventures with his two kids - he uses his leave to take them on these amazing trip during school holidays and cover for his DW who works more normal hours but has less leave. DB fits his cycling round the family, he doesn't use it as an excuse to avoid them.
Our best friends are big cyclist tourists, but neither of them would ever f*ck off alone for three whole weeks! They plan huge cycling holidays with trailers, panniers, tents - they've done this since their two kids were tiny, with baby trailers, then follow-me attachments for their bikes, and now the kids cycle independently with their own panniers.
DP and I were big mountaineers pre-kids, 6 weeks wild camping, hut hopping and peak-bagging was the norm every single summer, a two to three week trek would be nothing - it was bliss. Does either of us now demand 3 weeks off? No, we take the kids camping in the mountains, do valley walks, gaze longingly at the tops and very occasionally take turns to spot each other a big day out - or, if we can, sneak off for a couple of days when kids are at the grandparent's.
So I would say, it's not the hobby, it's the attitude. The question is, do you see that attitude changing?