OP, he wasn't running late, he was reducing the time he had to invest hanging out before you turned to the sex stuff.
This happens to women all over the country every day, it's nothing wrong with you or any one of us. Being treated with respect is a basic expectation, not something some of us deserve and others don't.
This one was always going to end in a ghosting either pre or post shag because that is what he is about. You could have been the Queen of Sheba and it would have been irrelevant, because he wanted to get laid. I'm sure by holding back you at least don't feel even shittier that he got his way.
The difficulty is that the good ones and the crap ones all dress the same, it's only when they start revealing themselves (turning up late like that, dick pics, "oh I couldn't help myself" etc) that you know and that's when you run. With the good ones, it all feels easy. The minute you find yourself trying to analyse what's happening that's when you know it's not going to happen and it's time to throw that one back.
And finally, please stop using language like frigid and prude, you do yourself and us all a disservice by using this language which is devised to shame us into bed. There is no right or wrong when it comes to choosing to have sex or not (assuming you're both consenting and not hurting anyone). Sleep with every man you meet or remain celibate for life, anyone who shames you for either can do one.