I think this is definitely a turning 40 "thing" - and the fact that you have both just waved your DS off to Uni. It highlights the very stark contrast between your DS's life (just starting and full of possibilities) and your DH's life (in his head - over).
As you had your DS young (and even if you didn't actually), I can see that now seems like a great time, to release yourselves of (some) parental responsibility and grasp a new lease of life. My own DH and I have said for quite some time, that next year we want to really start to travel a lot (youngest is off to Uni this summer, so nest will be empty).
Surely one of the benefits of having your DS so very young, is that now he's gone, you're still REALLY young! Don't forget, you didn't "live it up" when you were young, so neither of you have ever really had any freedom. You may not be that fussed, but clearly he is.
I cannot imagine, having an empty nest, but having my wings clipped because of an anxious Partner. I would find that irritating (sorry).
Could you have counselling/hypnotherapy to try to treat your travel anxieties?
Could you compromise and plan China for next year (then you'll have time to book the time off work).
Having said all of the above, the comment he made about not staying with you, if you were Pregnant, is truly terrible. 