My own mum said this to me a few times...but I just stick out my chin and told her I'd have managed just fine if we'd managed to have more than one. Just like every mum does. Now she doesn't mention anything about other children, just tells me I'm lucky to have just one. She was never maternal, despite having 6 of us!
I have sisters with children, 3 of them (one is childless, by choice, though why I felt the need to add that, I don't know). They all kept hounding me for when I'd have another, but they finally realised that my age was increasing and my odds, inversely, decreasing, so they stopped asking. They all now have grandchildren, thanks to a big age gap between me and my youngest sibling, so I have more in common with their kids than them a lot of the time.
It's often the snappers that gloat in the 'you'd never cope' argument. It's just projection on their part. They feel smug in knowing you may never be able to prove them wrong...and confident, despite evidence to the contrary, that they cope admirably. It's not worth getting het up about. And it sounds like you had a great birthday for your dc, which is all that counts.
And I'd defy ANY parent, one with 1 or 11 to not find parties stressful!