What you look like, what size you are is irrelevant.
He's admitted he's with someone he's not physically attracted to and is unapologetic about his words AND actions on the subject.
Does he take you out and about? Is he proud to tell people he's with you?
Because honestly it doesn't sound it and that being the case you deserve SO much better!
People are attracted to different looks for lots of different reasons.
One of my brothers best friends (very tall slim guy himself) LOVES tall, very full figured women. He's had criticism from arses for it, he tells em to wind their necks in! Both his wives have been this body type, when he split from his first wife (drifted apart) a dick he works with made a stupid comment about how he was now "free to find a real hottie" and when he started dating his second wife this guy was all "I don't understand why you chose her" she's funny, kind, intelligent, gorgeous, tall and very curvy. Bro's friend absolutely adores her and they have a great marriage. That's what matters - that they love and are attracted to each other.
Another has a thing for redheads (I'm a redhead and he asked me out a lot when we were younger but I felt it was too weird as I'd known him since we were at primary!)
Someone else I know is slim, petite, pretty and yet her husband gets bent out of shape if she even gains a few lbs, he doesn't treat her well and she's terrified of gaining weight. They've 2 children and each pregnancy she was on pins and needles about the weight gain and he did give her loads of grief about it and she was dieting practically as soon as she was home after giving birth! He is no fucking oil painting either! But he is a flirt and is disrespectful. Personally I think she should dump the arse!
As should you. It's not your body type that's the issue, it's his attitude to you that's the problem.
Everyone should be with someone who is proud to be with that person, who thinks they're gorgeous and an amazing person.
Whatever else I can say about my ex he would NEVER be like this. I was very slim and quite small of bosom when I met him which he obviously found attractive, but during our time together I lost weight becoming very slim at one point (size 6) and then later gained a heap of weight after having our dd. He still found me attractive and complimented me and talked me out of criticising myself too.
Now he cheated on me but here's the thing- she and I could be sisters physically! Including fluctuating weight. Whether someone loves you isn't just about the physical attributes they have (though physical attraction is an important part of sexual attraction it's not the whole equation). He's also pursued me for a physical relationship since our split when I've been at both my heaviest and lightest. Makes him a disloyal arse but he's not ever said or even implied (and believe me I asked!) that his cheating on me was anything to do with my physical appearance.
I'm not sure I've explained myself well. I hope you get from it that you deserve better.