Some people may choose to be late but a lot don't. You had it right when you said people "can't prioritise".
Prioritising things, planning and organisation are brain functions that vary between people. We're not all the same at that any more than we're all identically good at reading or maths or sports that require physical coordination.
Look up the long-term effects of traumatic brain injury (ADHD has similar symptoms). Things like depression cause problems with this too.
Even those people who aren't bad enough in this area to get diagnosed with anything still vary along a spectrum. We're not all the same.
To those people who'd end a friendship over this without a backward glance - would you do the same thing if you had a friend who couldn't read as well as you or who couldn't throw a ball as well as you?
I'm not saying people should be just let off any amount of lateness, but just bear in mind that if someone does have problems in this area, they know they're not being deliberately, thoughtlessly late, and so it may genuinely not occur to them that that's how it looks to other people.
They may just see themselves as trying hard, having the right aims even if they miss by a bit, and since they know they mean well they think it's OK, unless and until someone complains.
I should say I'm talking about people who are slightly late not so many hours late with no message sent ahead that it really can only be thoughtlessness.
There are people who are so bad at this sort of thing that to succeed they might need to spend the whole day getting ready for an appointment at 4pm, just like someone with dyslexia might take ten times as long to read a news article as a good reader needs.
Since everyone has a duty to try to be on time, then the people who are really bad at organisation and planning and time management have a duty to spend that whole day getting ready, if that's what it takes, or not make the appointment in the first place.
Once you work out that you need that much time, and that's OK, that's just you, then it's easier to put it into place. But most people don't expect you to need that much preparation time and it isn't always possible to get it given competing demands of work, family, kids and so on.
So it can become a process of muddling through juggling things badly and screwing everything up, feeling embarrassed about being late often but feeling powerless to fix it - because it really is not a case of just not caring.