The comment wasn't great but I wouldn't judge someone on the basis of a single sentence.
Is he usually a good partner, hands on, etc ?
I appreciate the "time is running out" comments may not have been very considerate... But if he wants kids he's probably getting desperate by now. I know I wanted children very badly and if I was with a partner who was "maybe, later, I'm nlt sure" while every month makes it less likely to happen, I'd feel... Strung along, I guess ?
This doesn't mean you should have his child. You should think hard about what YOU (not him) want. But considering your age I feel like you do owe him a straight answer. Will you try to have children with him, yes or no ? You've been four years together so you should have a clear idea what sort of partner he is and whether you feel confident he'd be a good dad, regardless of the occasional dumb comment.
If he's generally very hands on and this comment is the only thing holding you back it's be worth asking him to expand on why he wants kids.
Oh, and considering your career will be the one affected by a pregnancy, I would only marry him if he is better off than you financially.