I get what you’re saying, and a good teacher should be aware that a child might struggle with inference in a story, but shine in other areas. That doesn’t mean they won’t persist in also trying to develop their inference skills.
Subject skills are interchangeable. I get told things regularly like, ‘well they’re not interested in art’ or ‘they won’t need art’. We teach motor skills through art, we teach language and emotions through art. You don’t need to be artistic to enjoy looking at art. I have no idea what these children will want to pursue when they’re older. I want them to have a broad range of skills and experiences as a foundation.
My current (older) pupil who likes stories for much younger children, is very interested in coding and engineering and outperforms all other pupils in these lessons. Certainly I, and other teachers I know, do not put children on a shelf that matches the band of books they’re sending home.
Of course, I don’t know your child, your child’s teacher or their thought process My advice would be to address your concerns directly with the teacher. I would be very interested to listen to what your child is able to do at home and discuss how that compares to what I see in the classroom, or what skills I’m trying to develop. A good teacher would certainly try to find a wider variety of appropriate reading material.
I’d also take your son to the library and choose lots of books for you to read together - non fiction is great if he’s interested in them. Reading them together, particularly if its subjects he is engaged in, will encourage lots of discussion and will help his speech as well.