I'm puzzled by the scarf. So Rob found it at the Dower House while he was picking up some papers for Justin, meaning:
A) It was in the office/sitting room/kitchen
Or
B) It was in a bedroom and he was having a good poke around.
If A, so what? People leave scarves/gloves etc in other people's houses all the time. Lillian and Justin are spending loads of time together (openly) in her capacity as Social Secretary. And it's not that long since she was responsible for overseeing the redecoration of the whole place.
If B, surely Rob wouldn't want Justin to know that?
I can't work out what's supposed to be happening here.