Agreed! I have a PhD in Linguistics - everyone has an accent. You might speak with an RP accent (Received Pronunciation - i.e. BBC English), SSP (Scottish Standard Pronunciation), or other standard pronunciations for your country (American, Canadian, Australian, or South African, for example); but EVERYONE has an accent of some type.
I have a very neutral Scottish accent, from central Scotland, and as I lived in New Zealand for 3 years, I have flattened/softened some of the edges of Scottishness from it! My Dad is from South Africa (left at 13 years old, but first language was English, but spoke Afrikaans and a bit of Xhosa growing up), and my Mum is Scottish, so there were We also lived in Hong Kong for a couple of years as kids, and in the Netherlands too. My brother (as the youngest) sounds decidedly more American in some of his accent, as we went to international schools, but my sister and I had our accents more settled before we went there.
I have a colleague who has a party trick of guessing people's location histories, from their accents (they are a linguistic anthropologist), and he can definitely pick up any significant influences on accents. He reckons if you have lived somewhere, or with someone, for more than a couple of years, he can pick up on it. It's fascinating!