Hi MrsHeley
First off I think you can see there is a concensus here - he's only 4! So don't take this too seriously. You have by no means let him down. He's most likely much loved and is healthy & happy. You're off to a great start. YOU HAVEN'T LET YOUR SON DOWN! (Sorry MN for breaking rules on all caps - but I want to stress to MrsHeley she's doing all the right things).
Right - how to take this. Well this is the teacher telling you she's not happy with his progress, but not establishing how poor that progress is. She's involving you but possibly not going about it in the best way.
First of all you've no idea how 'bad' things are. It may not be that bad - just a few things to work on here and there to bring him back up to expected attainment levels. Trust me, having been repeatedly told all was fine only to discover that my DD1 couldn't read well, write well or subract at all by March of Y2 is much much worse place to find oneself. This is a school trying to ask you for some more help at home. You should be grateful for the honesty - it really is much better you know there are problems.
I'd arrange to meet with the teacher, if possible without your son and really talk this through so you understand the issues. Try to ensure the teacher understands you respect their professional opinion and that you want to work with the school to help.
Do remind yourself that if your DS won't turn 5 until late in the school year, or even the summer - then that can be a huge gap socially & educationally at this age. So do give him time and space. Now for the hard bit - even if you are stressed about his performance, you have to try to ensure that you don't pass that stress onto him. Start from where you are at - really get to know what he can and can't do - then you really can sincerely praise each little milestone he makes as he progresses.
Finally remind yourself that school isn't about producing millions of physicists - it's a great big world and there is room for all sorts of talents. I stumbled across this video at some point - it made me feel better, so might help you too. You need 20 minutes but it is worth it: