His comment is actually your get out of jail free card. It has woken you up to things are not as rosy as they seem.
You might be looking for a white knight, someone to repair you from previous experiences, but he is not a rescuer, he is an abuser. He wants to control who you go out with, where you go, he wants to go with you, and then he is starting to gently kick at your self esteem through your appearance before he eventually goes all out to destroy what confidence and autonomy you have.
The red flag is not a flag, it is a king sized red duvet cover of a flag. 26, you are in the prime of your life. He is middle aged, has lots of baggage, not even Clooneyesque, and he is going to get worse, in temperament and looks.
You need to not be in a relationship for a while. It's a big world out there. Expand your limits, do something that is not about relationships and settling down. Be free, be young. Time rushes past so fast, you are in danger of letting your prime years be taken over by somebody who has had his.