“Shining, a typical conversation would be
‘If you went in the office maybe a day next week that would be GREAT because I could have Rebecca and Davina round? They’ve been wanting to visit.’
‘They could visit, it’s not a problem.’
‘But we’ll be loud and disturb you.’
‘You won’t, it’s not a problem.’”
OP, surely at this point your next line is “Ok maybe it’s not a problem for you, but it’s a problem for me because we won’t really be able to chat and relax in the same way if you’re there working”. But you say you don’t want to be that direct as it would cause an atmosphere??
It seems like either a) he is a very literal type and if you say “I don’t want to disturb you” he thinks that’s literally all you mean, so if he says it won’t disturb him, the problem is solved.
Or b) he is deliberately being obstructive, and you feel obliged to make everything about his needs rather than yours because you’re worried about his reaction if you express needs of your own.
You won’t know which it is until you start being more direct.
As an aside, why would him returning to WOH have any impact on night wakings? You are both parents, you both need sleep, you share the night wakings.