I've had this problem and am still working on it.
Some of the reasons were:
Social anxiety and a strong fear of being early and having to make small talk with people I was afraid of (almost everyone).
OCD and feeling too anxious about every little thing which must be done before leaving the house, and which at that moment seemed massively important.
Both of the above fears would rule my actions, even above any embarrassment or feeling of letting others down. I already had low self-esteem and failing to meet everyone's expectations again couldn't have made it even lower.
And yes, at some times in life I've been late for important things as well as day-to-day things.
I didn't even realise for a long time that most people assume you're being deliberately rude or selfish when you're late. I had thought no-one would really notice or care. I remember one occasion where someone was concerned I hadn't arrived, and I was genuinely surprised they even noticed.
I never let anyone know IRL that I've had mental health difficulties. Perhaps some people might make allowances in a kind way, but it would seem a bit greedy to ask for this, and I don't want to be spoken to like a charity case, or patronised. It would be too undignified and it has felt "less worse" to let everyone assume I am just a rude, selfish person.
Of course, I've been angry and upset with myself many times. However, the real changes began when I finally found the right help and developed some self confidence.