The scarf was enough to get Rob's spidey senses twitching hmm, I wonder, and then go on a fishing trip, judging from Lillian's initial slight fluster that there was something going on and then pressing on with his insinuations. He can probably also guess that Lillian and Justin might be a bit at odds at the moment because of the whole job and amside thing, so she may be wary of just telling Justin about this conversation. The threat was implied rather than overt, it would be liable to just cause further tension.