and that I shouldn't have to stay late 'just in case' something crops up because my colleagues all decided to bugger off down the pub.
My company operates a flexible working policy and we have an informal "im leaving at x time" convo first thing. I was supposed to be out by 4pm for an appointment (hence getting in before 8) which I have now missed.
If you told your boss when you came in that you had an appointment at 4 ... then s/he should have ensured there were other staff available to enable you to go. If boss didn't do that then they are at fault.
However if s/he did ask someone to make sure to be about ...and they've just ignored her/him then it's the fault of that individual and quite reasonable for your boss to ask you to stay (assuming you're in the sort of job where someone HAS to be there).
When was the pub jaunt planned? If it was before your appointment, I also have some sympathy both on the part of the boss and on the part of the pub attendees that they didn't want to change their plans - I'm assuming the appointment was not something absolutely critical or you would have had more notice.