If the fling is several years ago, & they've 'been there, done that', & are genuinely now close, supportive mates, then YABU...
...unless she doesn't then extend that friendship to her friend's partner, ie. you.
If you & she aren't friends, is that because she's not interested in getting to know you (bad sign), or because you have always been a bit of her?
Actually, I'd be concerned by his lukewarm defence of her - he 'owes her a debt of gratitude'?
If his feelings for her are just 'old time's sake', & yet he stays in touch, that would bother me more (given you have made it obvious you'd rather he didn't), than if he were saying that she was one of his dearest, oldest friends, & as such he wasn't prepared to cool the friendship - which I'd understand better.