Right. Had you thought of downing tools, and saying that in fact you think you also might have an important lunch appointment tomorrow?
Tell him that you aren't fussed about dovetailing times, because you think the best thing would be that when he's back from his lunch, he can pick you up on the way home... and then you can deal with the removal together.
As he has a time frame in mind, and as you don't (because he has refused to tell you, presumably because he likes his wife to feel upset, humiliated and under extra pressure) he can call the removal men and reschedule.
Why on earth do you think, now, that the onus is on you to do anything less?
In fact, don't tell him you have plans. Just don't BE there when the removals turn up.
That's right. Just fuck off, like he has.
Teach him a serious lesson on how his future might start looking if he doesn't learn to start treating you with respect.