My ds was well below expectation throughout infants in what is a high-performing school. By end Y1 complaining "I'm bottom of the bottom group" and halfway through Y2 his teacher told us he'd failed to meet all his simple targets.
Then things turned round - he was already getting some support (had to nag for this) but he was lucky to have a brilliant Y2 teacher who spotted low self-esteem and gave him lots of confidence-boosting stuff. It really made a big, big difference. He's Y4 now and racing ahead - turned out he was probably 'late developer'.
Just keep encouraging him in what he likes and keep boosting esteem. Having good teacher does make big difference though.