He feels a "desire" does not mean that he should go ahead and do so.
One of the thing of learning to be an adult is when to say "no", and that includes moral decisions of this type. The fact that he can't do it, or held back from deciding this, is something that I would indeed consider and actually question. I was talking to my ex, and we commented on a friend of his. The guy out of sincerity and loyalty to his 2nd partner, went ahead and had a vasectomy. That is a very mature decision in my eyes. It means to me that he has decided that is his duty in his lifetime to be there for his 2 children. What he earns now, he gives some to his ex partner for the well being of his first child, and then some for his second child in his current marriage. He also married young too.
To be honest, I agree with your family and friends. I wonder if you could see yourself through their eyes and if you could be mature and see and make decisions which actually is at the core of you. He is 20 years older than you. Ask yourself truthfully if you are doing the right thing, and if he is also doing the right thing too, than to just going along with it. Even though his answer is a very diplomatic one.
It is innately for humans to have moral values, and moral decisions. If this question and doubt is inside of you, it must be there as a guide, then listen to it. If you felt that this is absolutely right and just, I dare say that you would not have to ask us for our opinions, and for validations and reassurances too.
I know that I personally would not date someone 20 years older. I did have ex-colleagues who are much older than I. When I socialised with them, I felt that was the "normal", but I also realised that in doing so, I miss out on actual learning and growing up too. Even though all my peers had such dramas. Learning self control and learning to go with the flow of what you should do for the age that you are is key.
Do not try and be one of these people who find common pain as part of the relationship. Or to let other people an opportunity that they can "do right what once went wrong" before and let them live their mistake through you now. It won't end in happiness.