OK, I reckon you've got two possible things going on here. I don't know which one it is, but I think I've played the part of your boyfriend in both these scenarios over the years.
First thing he might be referring to is that you're so quiet when he's going down on you. My DP is like this. Fingers, PIV, she's quite "enthusiastic", but oral she just lies there dead silent until she comes. And by god it's disconcerting. So much so that after about a year, I very rarely performed oral on her, because I thought it didn't do much for her. And for four years that went on because I assumed she preferred other things and she never asked for it. Until one day she cried and asked why I didn't want to go down on her. Turns out, she loves oral, but in order to enjoy it she really needs to concentrate, and that means silence.
Honestly, it's still disconcerting. I give DP oral all the time now, but it's for her benefit. Every other girlfriend I've had, oral was something I got pleasure out of as well. It was a turn on, feeling their movements, hearing the moans, that sense of escalation until she hit a peak. With DP, it's just repeat these actions until at some point out of nowhere she tries to pop my head with her thighs. It's the only time during sex when I've ever contemplated work, or the shopping. Don't get me wrong, I'll happily take 10 minutes of my mind wandering if it makes DP happy and the rest of our sex life is fantastic.
Possibility 2 is that he just doesn't understand the orgasm gap. Not properly. My ex before DP was like you @Thestoryof , in that she could only orgasm through oral, and sometimes it felt like oral was the only important part of sex for her. And frankly, it probably was.
Men get lucky, in that we orgasm easily, but conversely, that just means our orgasms are less impressive. An orgasm isn't the bit we're working towards, that we're hoping for, it's just the thing that happens at the end of sex. We don't have mind blowing orgasms, not usually anyway. No man has ever passed out from an orgasm, we don't lose the power of coherent speech, our eyes don't roll back into our heads. Not in the same way women do anyway. For us, an orgasm is just the slightly better feeling bit you work towards at the end because you know you're going to be too tired to enjoy sex if you carry on much longer.
Sex with my ex tended to follow a pattern, which sounds similar to what you have with your partner. Bit of foreplay, maybe a blowjob, I go down on her, she orgasms, we do PIV, I orgasm. And it felt a bit, not monotonous, but static. It felt like she wasn't interested in the foreplay, she wanted to get to me going down on her. Like that bit was for her, and then the PIV was for me.
With DP, and previous girlfriends, sex feels much more asymmetrical, and yet more like we're both enjoying all aspects of it. There's no following a script, a set of tickboxes to make sure we both orgasm. For DP and I, there is no pattern for our sex life. Maybe tonight she'll give me a blowjob and then roll over and go to sleep, or I'll do the same for her. Maybe it'll just be a bit of mutual masturbation, and we'll both orgasm. Maybe instead a 69 and some PIV. Maybe we'll have a full 4 course banquet and do absolutely everything. Or maybe we'll just have a few abortive attempts to fool around in between pillow talk before deciding we're both too knackered and decide to pick it up in the morning.
With my ex, the fact that she required something so specific to orgasm, made her that much more focussed on it, and made our sex life that much more restrictive in comparison. To be honest, we didn't go out long enough for it to become a problem, but if we had, it might've done.
For whatever reason, it sounds like you only managed to have half a conversation with your partner about this. You got the criticism bit without getting to the constructive bit. You need to talk to him again, and find out exactly why he thinks this is an issue, and what his ideas are to fix it. Maybe it's something simple, maybe it turns out you're completely incompatible. The only way to find out is by talking more.