Because just because you know someone, it doesn't ever mean that you really know them.
Or have seen what goes on when they are with other people
Or behind closed doors.
It's been used for years to allow people to get away with all kinds of things. Some truly awful things.
If you are using that defence to allow someone to get away with something, then you need to think about how that makes the victims feel.