If you don’t want to, that’s good enough.
The thing for your DH to bear in mind is that (and obviously I hope this doesn’t apply to either of you) your body/health can change quite a lot between your 40’s and 50’s. I developed arthritis when mine were 9 and 11 and now that they are late teens, it often means I am much more restricted in what I can do.
The thought of being where I am now when my kids were still in primary school and I was still expected to be running around, climbing, chasing etc, makes me shudder. Mine are the age of your DD so although we enjoy going out, cinema trips etc, it doesn’t need me to be so physically fit.
Also, you are young enough that you & your husband can really enjoy travelling/hobbies/sports/whatever you enjoy and have some fantastic times ahead of you.
Is it possible your DH isn’t just having a wobble at seeing your friends with kids at the little, cuddly, adorable, Santa Claus stage and just misses it, rather than having a genuine desire for more children? The