Looks matter of course. But often when you meet someone who is gorgeous and get to know them, they can become less physically attractive because of their personality. And the opposite is also true. You meet someone whom you think is just plain, and after getting to know them, they become more attractive.
I think this is what "loving the person inside" is about. If you come to detest someone, they do not remain good looking.
Your sample size is three men. That's not really indicative of all men. Just those three. We've all dated twats. Sometime breaking up hurt, sometimes it didn't. But you move on and learn what to NOT date. As Cogito suggested, make a list.
Maybe you are putting attractiveness or forcefulness or some other twatish quality above a sense of humor or sincerity. Abusive jerks are out there, they'll find you if you go into a relationship willy-nilly.
Relationships are hard work, but take some time off, find yourself, then go find a partner.