OP, my DH said something similar to me once - different circumstances and context, but I 100% understand why the comment hurts.
I'd been unemployed for a while, no luck finding another job, and was struggling badly with depression - not being able to get out of bed or brush my teeth type of episode. At one point I happened to have a very busy day with a lot of errands, and I was absolutely knackered by the end of it, to which he said something along the lines of "well, it's no surprise you're tired, you're not used to working."
It was, objectively, true - I was probably knackered because after several months at home, I was no longer used to having fulll days.
But at the time it really made me, as you say, hit my limit. It hurt a lot because I wanted to be working, and because I'd been trying really hard to apply for jobs with no success, and it just felt like such an unfair and insensitive comment - even though it wasn't meant that way.
In hindsight, I can see that it was obviously tapping into my own insecurities around not working and being able to contribute to the household, and while phrased a bit clumsily, it was a neutral - if not supportive - comment.
Is there any chance there's some of that going on?