Don't give up on him, blame him or in any way let him know that you feel this way. 6 is so very, very young - don't forget that in many countries, children haven't even started formal education yet. If he continues to struggle, you may need to decide it is not the right school for him - but because they are failing him, not the other way around.
My DB struggled all the way through school, in a family of pretty high fliers. He failed most of his exams and was made to feel like shit. In his 40s, he recently got a First in his OU degree, whilst working full-time (the only one of us to get a first in the end!).
My DPs basically thought if him as the 'dim' one, so didn't question the school, which in retrospect was the one at fault, not my DB.
Discuss positive, constructive strategies for helping your DS with the school, tutor him yourself if you can, or get him a tutor to improve his confidence and skills if you can't.
But whatever you do, don't label him or give up on him!
There are many ways for kids to be bright, and kids develop at different speeds. I vividly remember my DD telling me she 'couldn't' read proper books - I told her she could and she would - and so she persevered and she did.
Your son needs you to believe in him now more than anything, even if he doesn't believe in himself (and it sounds like his school may have dented his confidence, unless they're a lot subtler to him than they are to you). EVERYONE is fantastic at something. Find what he enjoys and encourage him, separate the things he finds hard into little bits and tackle them one by one, all the time believing in him.
He'll be fine. Honestly.