Hidden disability related - though I have been late for work more times than is really acceptable and I did used to be late for school every single day, I could see how this would look like a discrepancy as I am normally on time for work, flights etc yet before I had better management systems in place (and still, sometimes) have tended to be/am late to meet friends.
Also, somebody asked if an often-late person knows the calculation idea, counting backwards etc works why not just do that? For me anyway -
Work and school tend to be the same time every day and hence it's the same routine to get there and the calculation for what needs doing only needs to be done once. Plus if it's wrong it can be adjusted ready for the next time. But mainly the fact that it's already been done/calculated, means I don't need to think about it and calculate it again.
I also learned when I was in college to incentivise myself to leave early by aiming to arrive on the bus one earlier than the last possible one. By doing it this way I allowed myself to buy a coffee in the canteen or similar before my first class. This is a philosophy I've carried on, however, 90%+ of the time, I will miss that earlier bus and arrive on time (but "late" according to my own plan).
For something huge like a plane or train for a long trip I allow myself masses of extra time - like hours. And again still end up cutting it fine sometimes.
But anyway - I don't tend to do this when I'm meeting friends. Mainly that's because it doesn't really occur to me. Nobody puts as much preparation into planning to meet friends for lunch as they do to catch a plane to another country. I do more recently tend to stick the details into a journey planner and work out the times I need to leave, but I probably wouldn't think to do much more than that.
I do have a bit of a tendency to disbelieve myself, too. I'll calculate how long it will take to leave and then when I need to start getting ready and then I'll think incredulously "But that's a whole two hours before I need to even be there! That can't be right." The trouble is it is normally right 
I'm getting much better at overriding that script, but it's taken me years to get to the point where I'll grumpily go along with it rather than just thinking nah... and ignoring it.
The other thing is that it takes a huge amount of mental energy to do the whole counting backwards, working out how much time I need, kind of thing - and while that mental energy is justified when it is in relation to something which will be repeated many times and hence is valuable many times over, or for something which would be utterly catastrophic if it was missed and which warrants taking the best part of a day to deal with, a casual meet up does not fall into either of these categories and I do not usually have an entire day to write off just to ensure that I meet the friend on time. If I did this for every thing I would never do anything. Having to put massive amounts of preparation into something does make a difference to the overall difficulty of it - just think of how difficult it is to get out of the door with a pukey newborn or a potty training toddler vs how easy it is to get out as a single adult on your own.
And while I understand that it's not a totally automatic, easy, effortless process for everyone else, someone else commented something like "I can't imagine having a shower and not realising that it would take time to dry my hair. Surely that is completely obvious?" - well, no, and that's what I mean when I say things aren't automatic!
Then the last problem is that yes - in fact I probably could channel more of my energy, time, mental health etc into ensuring that I'm always on time - but being late is not the only problem that my invisible difficulty bestows upon me. In reality there are several areas where I fall down - cleaning up after myself, keeping promises (to myself or others), being consistent in parenting, working, self-care (eating, sleeping, hygiene), keeping track of personal objects, maintaining positive relationships, and so on. I could meet normal standards of any of these things by expending a huge amount of effort and energy into them but it is physically impossible to keep up with all of them (I don't have the energy nor hours in the day) and therefore I will invariably aim for a "good enough" shot at everything I can, which means I fall behind exacting standards most of the time. I can't do it all - but it's not selfishness or lack of caring, and it's quite distressing to think that others perceive it this way.