I went our for lunch with a couple of guys today. The three of us are all different nationalities and all speak English. The two guys both have the same first language however.
Several times during the meal they started conversing in their own language while I sat there without a clue what was being said. We were the only adults present.
Part of me thinks that maybe sometimes it's easier for them to speak their own language and if they wanted me to know what they were saying they would have said it in English and the other part thinks it's quite rude and excluding - almost like whispering.