I also want to know if the baby cried. If he did this hard enough to make the baby cry, and still didn't notice, then I would be very worried, yes.
If he did it and the baby didn't cry, yes, I sympathise with you and I'd also have had that horrible 'heart in mouth' moment where you can't believe someone would do that. But I would say it's partly an instinctive overreaction rather than a reasonable one, if they baby did not notice.
FWIW I'm the non-birth mother and when my DD was very tiny, my DP was very concerned about how I was around her. I remember DP being horrified that I carried DD in my arms to the corner shop, rather than in her pram, and also when I made up bottles while holding her. I figured that I needed to change my behaviour as much as possible so DP wouldn't worry, and I did, but we still had issues where I realised DP was concerned about something I had not seen as potentially dangerous or worrying.
I still have no idea who's right and who's wrong. DP and I have discussed it and, now we're out of the early babyhood stage, neither of us is very sure what was going on! DP had PND and was really anxious; it's perfectly possible I was too blase.
The crucial thing is to talk about it.