Why would that be scary, unless someone has a huge ego?
I don't think you need a huge ego to be scared when faced with the utter futility of your own existence. Some sort of ego, yes, but we've all got one of them - pretty hard to function without one I'd have thought.
You don't find the fact that you are running around 'doing the right thing', striving and struggling to make the world a better place for your kids, and those you love, and it "don't (as a great philosopher once said) amount to a hill of beans in this crazy world" even the tiniest bit scary?
I'm 60. My Grandfather spent 60 years on this miserable planet and died about 57 years ago. The only thing I know about him is a (possibly grossly exaggerated) story told to me by my father (also now dead). As far as I know that story is the only thing ANYONE knows about him apart from parish records and census data. When I'm dead in ten or twenty years that will be gone too. I doubt if my kids will be bothered to remember it. Means even less to them than it does to me.
Everything most of us have ever done, said, or created will be gone in a couple of generations. Forgotten. Just as if it didn't happen.
Of course it's scary. That's why people make up comfort-blanket stories like 'heaven' to keep the dark at bay.