Genuinely not wanting to generalise, but I have found that those saying 40s is too old are those who had kids in their 20s or early 30s, and felt/feel old in their 40s. Those of us who didn't have kids til our mid 30s and early 40s (me!) DON'T feel 40 is very old. I am early 40s and have a toddler and feel that I totally keep up with the younger mothers around me, often better. And actually, the vast majority of girls I went to school with all have kids around the same age as I do.
Also those saying don't go for it were mostly lucky enough to have kids when they were younger, and don't understand not having a child.
Regarding your relationship. It's not ideal. But, if you both want a child, that child will be well-loved and provided for. I think you need to be honest with your guy, about what he is signing up for (possibly co-parenting rather than a family) and if you are both ok with that, go for it, and good luck.
To add, I had two easy pregnancies, the toll on my body is negligible; I was very fit before my first. Look after yourself now if you aren't already - gym/exercise lots, eat well (ideally whole foods plant based, very little or no processed crap) and hopefully it will happen for you.