Perhaps you could clarify with the teacher. Being securely within 'expected' but at the same time 'behind' and in 'bottom group' doesn't obviously add up, so worth finding out what is going on. Personally I would discount most of what you hear from other parents though.
There are so many ways communication can go wrong at parents' evening.
OP, you sound really convinced that the teacher actually said your child was 'behind', but that sounds unusual. I would expect the teacher to comment on how your child is progressing with regards to where they were when they started, and with regards to EY goals, but not on where they sit within the class - this is quite meaningless (as cohorts can vary hugely). Are you sure the teacher said 'behind', and was it with regards to phonics, reading (e.g. comprehension), 'English' overall...?
Just last week a friend told me she'd missed her reception child's parents' evening, her husband went instead and came home saying how the teacher had said the child was struggling in all areas really. When she went to talk to the teacher about this a couple of days later, the teacher was really confused, and finally clarified that she had just listed the next steps but that in fact the child was really quite advanced. Sometimes we don't hear what is said!
Miscommunication aside, I see several ways in which reading yellow level books at this point in reception could lead to being 'bottom group for reading'.
Firstly, as you say, it might be a high achieving cohort. Is it a 'selective' private school by any chance? Or a very hard to get into, fully upper-middle-class leafy state school? I find it hard to imagine that more than a few exceptional cohorts in 'normal' schools would have 25 or more children per class reading beyond yellow level at this point in reception. And even then, your child is clearly not 'behind' as that would be with regards to expectations, not with regards to other children; and as he is solidly within 'expected', it doesn't really add up.
Secondly, your school could be using reading book levels differently than most. For instance, they could be giving books home that are a lot harder than the child's actual reading abilities. Which would mean that at your DS's reading level, he would be reading red or pink level at most schools. That would tally up with being bottom group in some high achieving cohorts I guess. And could still put him solidly within 'expected'.
Thirdly, might it be a distinction between phonics and 'reading'? Schools are meant to provide the children with books at their current phonics level, NOT at their current 'reading' level. Perhaps your school is actually doing this? So his phonics is pretty good, he's been taught all the phonics he needs for yellow level books (provided they are from a 'decodable' scheme), so they give him those books. But he struggles with the 'reading'... e.g. he doesn't have any stamina, loses track of the first words of a sentence before he reaches the end, doesn't understand what he's read, isn't able to answer any questions about what he's just read, ... All these things could mean that he is not yet a strong 'reader' although his phonics ability is right where it is expected to be.
As to if this will affect his targets, expectations. It shouldn't. If what you understood is correct, he isn't 'low achieving', he is exactly within 'expected' (just that some other children are currently a bit ahead). So if school will base their expectations of him on his current results, they should have perfectly average expectations for him, not 'low' ones.
And most schools with have a lot of experience with children 'clicking' with reading at some point between YR and Y3 or so, and then it all evens out, so they won't be thinking him a 'low attainer' but rather just that he hasn't 'clicked' yet.
If he is indeed at expected levels, but bottom group in a high achieving class, then he might benefit from extra help that he wouldn't otherwise get, being average; and from the fact that teachers won't have to always orally explain all tasks to all children (who then forget and need to be told again), as all children can read well.
If you are going to be concerned about something in this situation, it is perhaps his self-esteem. He might be aware of being in 'bottom group' but not that he is bang on track and doing fine. Depending on how school handles this ability grouping thing, and how nice (or not!) the other children are about it, this might impact him quite badly. That's what I'd keep an eye on.
After clarifying with the teacher what is really going on.