I might be overthinking this a bit, but here is my question:
A very close friend of mine and DHs lives in Australia. The three of us go back 25 years - we met at university. As he lives so far away, we don't see him very often. He is coming to the UK unexpectedly for a week to sort out some stuff with his ageing parents. They live about 3 hours drive from us. At this kind of notice, we are unlikely to get babysitters for our kids, so thinking DH and I will go over and see him separately - as it's quite a long way and we'd have to go in the evening, it'd probably make sense to stay over. Would you be uncomfortable with that idea if you were me/DH/friend/friend's wife (who is staying in Aus with their kids)?
In the interests of full disclosure, friend and I had a brief relationship when we were at university. It ended amicably when we went our separate ways after graduation. I don't know if that makes a difference or not.