I am going to take a slightly different approach as I am assuming he is older, perhaps divorce previously or at least long term relationship split up.
I think definitely yes saying no to seeing him today is right as it wasn't an early breakfast and you should be making an effort, but it was Frida last night and he might have got carried away in the pub with friends, and when you are single older friendships are important so dropping a, perhaps, evening in pub where he is being social, thinking, it's fine I'll be up by 10am, as in I won't feel that bad tomorrow it will be ok. Have you never had to bail because a previous night got out of hand and you had every intention of going bowling after lunch the following day, you think it will all be fine in your drink Friday night little mind, you are basically acting like you are 20 and can bounce back after a skinfull. And it all goes wrong. Well it could just be this.
If you like him OP and he apologies, and you calmly point out, if he attempts to make another arrangement that you are not going to assume it will happen and will clearly have backup plan eh but are willing to give it one more go! And he then makes every effort to get to where you agree, honestly I could let this go.
But acting huffy and moody about it is not a great look.
So either forgive, not forget, but water off a duck's back, and make another date, or bin him off.