I have a friend that grew up in Ireland and moved to Austria when she was 19. She does not have an Irish accent anymore and people like you assume all the time when she comes home that she is not Irish.
By you explaining about being bullied etc it should give you a greater understanding but it clearly doesn't.
Your example of the man you spoke with, being French and having no trauma is ridiculous. You didn't know if he had experienced trauma did you? Also it is extremely difficult to detect accents, yet you are telling me you knew he was from France.
I have a Moroccan friend that speaks French as it is widely spoken in Morocco. French is her first language. She speaks Arabic to her Dad and his family and French to her mum and whoever else is a French speaker. Would you have made a wrong guess that she was French and asked her what part of France she's from. She actually has spent most of her life in Germany and also speaks fluent German.
There are roughly 28 countries that speak French, in alphabetical order: Belgium, Benin, Burkina Faso, Burundi, Cameroon, Canada, Chad, the Ivory Coast, the Democratic Republic of the Congo, Djibouti, Equatorial Guinea, France, Haiti, Luxembourg, Madagascar, Monaco, Niger, Rwanda, Senegal, Seychelles, Switzerland, Togo and Vanuatu. numerous French speaking countries. He could have been from any of these countries.