I remember your previous posts, OP. He's gone to live with his Dad, is that right?
How do you feel about him possibly having a party or leaving the house unlocked while you're away? Would you be OK with that?
I have to agree with your DH. It sounds highly likely that your DS will use your place for mates/leave it unsecured whilst you're away and I would definitely want the key back.
But at this stage, as you have already asked and DS has refused, I'd be changing the locks. I know you say DS wouldn't be organised enough but if he thinks it's fine to use your home without permission in your absence then I am sure he'd get it together to take a copy of the current key.
If it means your DH (and you) will not be able to enjoy the holiday because of worry about this then surely it's worth it?
You don't need to say anything to DS about it. Just let the matter drop, change the locks and enjoy your hols.
It's hard to imagine why he would need a key for emergencies. Sounds like a poor excuse. At 22 he should be resourceful enough to cope or turn to his Dad/friends etc in your absence on hols.
As for the emotional side, you really aren't losing him. Your relationship will change and grow in different ways as he moves through life. I'm sure you have told him you love him and are there for him. You're not doing a bad thing.