DS1 has just started CP, so I have been through this, though some circumstances a a bit different.
Canteens are really noisy places, as they are usually overcrowded, in rooms not designed for the purpose and the staff are recruited by the mairie. Childcare and psychology skills are not the main criteria in recruiting them, in fact no skills of any kind enter into it, IM(cynical)O. They yell at the kids to be quiet because the noise is unbearable, but their yelling doesn't help and stresses the kids, obv.
In PS, I hadn't planned on DS1 eating in the canteen, he was meant to go to his assistante maternelle for lunch and the afternoon, as I couldn't envisage such a long day for a 5yr old. That fell though when she became ill, but by then I was almost on ML and brought DS1 home for lunch when I could.
In MS, DS1's teacher took me to one side after the 1st day and suggested that, if I couldn't take him home myself, I might try and organise a system with another mum. We did that for a while: I had her son for lunch one Mon, she took DS1 the next. I realise this isn't an option if he knows no one, although actually one mum did put a notice on the classroom door last year saying her daughter was in morning and afternoon garderie and the canteen, and could anyone help them out by taking her home with them every now and again because she wasn't handling the exhaustion well. Maybe worth a try if you can't get time off work?
FWIW lots and lots of people here seem to wangle it so the get their kids home from school. They seem to fall into 2 distinct socio-professional categories, et either end of the spectrum, with the middle-class professionals being the ones whose kids can't do this.
I am lucky enough to do a job where I work at home one or two days a week, and I'll continue to have DS1 home for lunch once a wk as and when I can. I do think he is less aggressive and crotchety on the days when he's not eaten at school.
He never mentions not wanting to eat there. I am the one who says I'll get him for lunch on such a day, and he always looks pleased (even when it means missing choc. mousse).
There is no feeling as awful as going off to work knowing your child is miserable at school, so I agree that if you can find a solution, it'd be best all round.