Small kids do indeed sometimes develop (what seems to us adults) odd aversions!
Reframing the trigger in a different, positive way is best, but not possible always to pinpoint exactly what that cause is. Especially with pre-verbal children. As you've found, OP.
It might be more the BIL's home environment (scene of [understandable, post-breakdown] unhappiness/ tension) your DD's struggling with?
Must be very stressful for you and your DH seeing him in a bad way: perhaps she's particularly astute at picking up that happens before/ after visits there, and associates him with that?
PP's suggestion to neutral, outdoors walk and distractions a very good one. Might break the cycle.
An anecdote to give you hope:
My cousin always fled screaming in terror from our (exceptionally gentle and kindly, but very tall) elderly relative. He was 6ft 5, though rather hunched over by then...through no fault of his own, I suppose he must've appeared to loom down towards tiny people!
Every visit for 2 or 3 years was tricky, as this poor chap felt (wrongly) his inexperience with kids was to blame. Various approaches were tried (no attention or fuss made of either, him remaining seated, no eye contact until she initiated it; tempting toys to share with her etc. etc.) to no avail.
This sweet man took it upon himself to read through my (by then) 5yr old cousin's vast collection of favourite books, in the hope they'd share common ground to talk about. He discovered Roald Dahl and decided he would 're-brand' himself as The BFG 
Curiosity got the better of her, when this proper, RL grown-up nonchantly offered her a glass of Frobscottle, and reassured her "I is a nice and jumbly Giant"...
Took a while but they did become great friends.
Family legend has it that she even asked her parents to insist he came to stay, so she could take him into school for "Show and Tell" 