I can personally attest to the fact that more lovers does not equal a more experienced lover themself. I'm 32, my SO is 35, I've had 6 lovers (he's the 6th) and he's had a substantial amount more than that - I don't know exact numbers because I don't ask questions if I know the answer will hurt me and the answer is irrelevant to our current relationship regardless. However from talking and sharing stories over the years I can guesstimate that it's 50 at the lowest, 100 is probably closest to accurate and 150 would be my highest estimate.
I was in the same position as you when we starting dating OP, the more I learned about his past the more self conscious I got. Then something happened that changed my thinking - he told me he'd never had an orgasm from getting oral sex - which I thought was a fib at first, but then after trying multiple times the way he said that he liked it with no orgasm, I believed him. Now the type of blowjob he wanted and said he "loved" was literally porn style, down my throat, eyes watering - the whole bit (and fyi, he didn't thrust into my mouth, I had 100% control, I was just trying to give him what he "enjoyed").
Anywho, one evening, I asked him if I could go down on him, and he started with the "You know I can't get off that way... and I hate how it hurts your throat the next day" I said "I don't want to give you THAT kind if blowjob. I want to take my time and make you feel good" he seemed hesitant at first, but after 30+ minutes of softly kissing and licking all of his lower body (tummy and thighs as well as his phallus) he was like "Babe...I think I'm going to... I'm going to..." and that was that. Afterwards he was like "I had no idea I could have an orgasm from getting head, you're amazing!" It made me feel much better, and since then we've had conversations about the fact that since all my lovers were long term relationships I got the time to learn their bodies and I got the feeling that since he had few relationships, mostly one night stands - that he'd never be able to get to a level where you learn your lovers body from head to toe. He tells me I'm the best lover he's had and I believe him (and knowing that he's had quite a few, that actually boosts my confidence now).