If you were with two other people, say Mary and Bob, and Mary said 'Bob' at the end of every remark would you assume she was addressing Bob and not you?
No other cues either. Mary doesn't particularly look at either you or Bob when talking.
AIBU to not speak when Mary says 'Bob' at the end of every remark?