I had my first at three months short of 43 and my second at three months short of 47. They were both IVF babies (second one was a frozen embryo from when I was 41/42). I would have preferred to have them earlier (tried for 10 years) but now I am 56 with a 13 year old and a 9 year old and things are fine. I think my children keep me active physically and young in outlook!
I am so glad I have them but there are down sides to consider - risks of possible disabilities being one. How would you manage if this happened? We were prepared to take the risk but we had to think it through and it was hard. As it is, one of ours has some additional needs - possibly related to our age - but, there again, it could have happened even if we had been younger parents.
In addition, although our finances are not too bad, the prospect of university fees etc mean that we are going to have to keep working for the foreseeable future.
Another issue is family network. Grandparents of 70 plus are unlikely to be able to help out that much and, in fact, we have been dealing with the challenges of elderly parents and young children at the same time. Any siblings you or your DH have are likely to have older children and not be so interested in your younger ones. Likewise, cousins may be quite a bit older than your children and not really be companions for them. However, there can be big age gaps even for younger parents.
It is true that the child is more likely to lose one or more parents at an earlier age but life is unpredictable and parents can also die young. You do need to make sure though that you are able to have life insurance and think about who might care for the children if you and your DH were unable to do it.
In my experience, I would say that your DH's views are the main obstacle but that you have to think through all the possible implications and whether you can deal with those. Also: be honest with yourself in terms of how much physical and mental energy you currently have.