I really hope you're doing the right thing, but I'm concerned on several points.
You said :
"pretty sure he wants me as a sex buddy because he can't get it anywhere else."
If that's the case, he's going to try hard to keep you around. Of course he doesn't like it if you talk to other men. He's afraid you'll meet someone who cares about you and wants you for you, not just as a sex buddy. Then he gets dumped and he can't get it anywhere else.
Even if I'm being harsh on him there, it can't do your self-esteem any good to be thinking that way about him. It sounds as if you think you can't do better than him - an older man who has made the choice to live a single life (no issue with that) and who can't find anyone to have sex with.
On the other hand, I think it's already more than just sex for you. If it was just sex, you wouldn't have needed to be analysing his behaviour, getting annoyed when he takes 4 hours to reply to a text, and feeling upset when he sent you that text the other night saying he just wanted sex.
After my divorce me and my 'fuck buddy' or FWB or whatever label we want to give it, went all round the houses with each other. It went on for years too, on and off, not just a few 'fun' months. We both ended up confused, feeling on the end of mixed messages, tried to stay away from each other but couldn't etc etc. I'm not going into the whole story of it because this is your thread, not mine, but my point is that it did nothing to help my self-esteem or heal from the divorce. It was a mindfuck. Yes, I know, not everyone's the same as me, but you're coming across as vulnerable right now, understandably, and I'm not sure this is good for you.
After a divorce, or any major break up, we need to do the things that help us to heal and move on to a healthier relationship than the one we've left. Improving our self-esteem is a big part of that. I'm not sure this fuck buddy arrangement will help you to do that.
Posted with only good intention, sorry if it's harsh. 