Some people really care about it. It doesn't really make sense to me, but someone put it to me like this "would you be happy if someone wore hot pants and a crop top to your mum's funeral?" (My mum died young and unexpectedly).
There is no logic to it. Would my mum have cared? Probably not. She was really funny and a "cool girl". Would I think about this (hypothetical) hot pants and crop top person "there is something wrong with that person and I wish she would get out". Yes, I would, being totally honest.
So although I don't really care about the white at weddings thing, I can empathise with people who do and therefore just steer clear. Some things just aren't considered appropriate for whatever reason and there doesn't have to be any rationale behind it like "oh no you might get confused as being the bride".
I can't say I'd have noticed at my own wedding though and wouldn't have minded the op's dress at all, so think it's down to the couple / other guests really and probably varies a lot from wedding to wedding.