He says he doesn't think bad things about me and that he doesn't think iv 'been' with everyone but it plays on his mind
He's lying. He thinks you've had it off with every man you've met, including the butcher, the baker, the candlestick maker etc, and that you'll continue to do so with them and with any new men who come in your orbit because he sees you as a bitch on heat - gagging for it with any man you look at, or who happens to look your way.
This scenario never stops playing on his mind because he doesn't trust you, or any other woman for that matter, and the reason for this can be found in his past way before his ex allegedly 'cheated on him twice'.
This deeply insecure man has no love for women. He resents what he perceives to be the 'power' women have over him and he 'breaks' the women in his life so he can gain power over them. From your OP, it's clear he's already begun the process of breaking you.
Don't make the mistake of thinking that, given time, he'll come to trust you, know you'll never hurt him, that you'll never be untrue, you'd never look at another man, he's your soulmate, you are 'meant' to be, 'love' will prevail, and other equally misguided sentiments.
If you stay with this man it won't be long before he becomes violent towards you - most probably just a slap or two first and he'll be so sorry, he'll cry real crocodile tears, he'll say he didn't know what came over him, it's because he loves you so much he can't bear the thought of losing you, and he'll promise it will never happen again... but of course it will and, before you know it, your life will become a living hell which, in turn, will adversely impact on your ds.
Don't waste time running - jump in the nearest car and put your foot down hard on the gas. The hills are that way >>>>