Sperm quality decreases (regardless of how old someone looks) as men age.
So what happens if you have a baby that might not be what he is looking for?
You aren't married, he walks away, you are left literally holding the baby, can't progress with your career or even keep it, no financial obligations?
Reality is you are late 20s he's 41, that is a huge decade in terms of fertility between you.
If I were you OP on this fact alone I would give some consideration on whether he is RIGHT FOR YOU.
Me & H met early 30s and to give you an idea we both gave ourselves until an outside age of 38 to get pregnant with any children we were going to have, due in part to the decline in genetic material from both males and females.
He is in too much of a rush, he wants a baby, he thinks and you are now some sort of box ticking exercise. And yes the age difference will be more apparent as you and baby grow.
Say you get pregnant in a year, he is 42, say he is 43 when baby is born, add about 5 years on, he is 48 at the beginning of the school gates, he will be 55 when, now teenager finishes secondary school, 57 when they start, if they start university. 61 perhaps when they finish uni, there is still potential help with house deposits, when is he planning on retiring, is he wealthy enough to be able to support all this providing his health maintains.
I know people do this at these ages and later, and yes it usually works out, but there are what 12 years between you two. Once you throw a baby, and these are just numbers for baby 1, what if you want your child to have a sibling, then add at least another couple of years onto these ages.
And he wants you to commit to all this without the security of marriage?
The baby bit is easy, it gets a lot harder before it gets easier, and it never gets less expensive.
Fuck that OP. You are at two different ends of a decade.