Jesus. I can imagine the lurching sensation in that moment, when your sight is suddenly knocked away from you. I feel the same vulnerability as you do, if I have to take my glasses off in public. If your uncorrected vision is similar to mine or worse, it's practically useless for safely navigating any environment or interacting with other people, and as we're usually well corrected by glasses, we've no experience of getting by without much useful sight.
People often can't seem to grasp how little I actually see without glasses. Maybe I explain, maybe they're surprised or disbelieving at how bad it is and that glasses can fix something so bad — and ten minutes later they're expecting me to see their facial expression from 3 feet away while I'm cleaning my glasses. So it doesn't surprise me that he didn't really get how shaken you must've been by almost losing your ability to see (temporarily), especially if you haven't had reason to explain to him how poor your uncorrected vision is — and this thread is ample evidence that many people have no idea it can quite reasonably be the financial equivalent of knocking a brand new decent-spec phone out of your hand (except that, in the case of a phone, you could replace it with a cheapie while you save up).
I can't remember whether you said earlier what you wear glasses for, but mine is mostly shortsightedness. And about 1 in 4 adults have some myopia/shortsightedness, so I think part of the problem is that people think they know what it means: fuzzy distance vision. When you wear glasses for shortsightedness, people will naturally assume you only struggle with distance vision — can't read distant signs, leaves on trees blur together, better pop your glasses on so you get the full benefit of your new 4K TV… For those people, losing your glasses is an irritation, because you'll struggle to follow the sporting event and might not be legal to drive, will be squinting at stuff until you get home and dig out a spare pair, and you'll have to buy new glasses (for a reasonable price).
But 1 in 25 people have high myopia (over -6D), which starts to seriously affect your ability to function without vision correction, (and the cost of the glasses). I'm at about -11D, plus -2D of astigmatism (I mean, pretty bad, but nowhere near the most severe myopia), and I can't focus further away than 10cm. Annoyingly, at 10cm, the images from my right and left eye are so different that they won't mesh with one another, so I can only look with one eye at a time. Luckily I can get very good correction with glasses or contacts. But if what happened to you happened to me, I'd have been quite upset, at least I'm part because it's a reminder of how precarious my day to day function really is, and how easily I can be made completely helpless.