I believe the custom was instated so that the ladies in waiting could always be easily identifiable. I don't believe Daisy has any worries about being upstaged.
I don't know what to think about Fred and Mary tbh. Did he have an affair? Who knows. But if he was stupid enough to show himself alone in public with an affair partner then he's really, really stupid.
However, if I were Mary and let's say I knew without a doubt that there was no affair, I'd still be absolutely livid at the humiliation of it all. I'd be so angry with him for putting me in that situation! So I'd be just as cool as her in the circumstances, whether or not anything had happened with another woman.