WHY are you keeping this a secret? How OLD are the pair of you, this does (please forgive me) make you both sound REALLY young and bordering on silly.
are you both free to date? above the age of consent?
Who cares what other people think about whether or not you are going out or not? what business is it of theirs? What do you care about what others think of who you are or are not going out with.
That's one issue.
The other, more important issue is that AS A FRIEND, this bloke saw you visibly shaken and upset and didn't bother his arse to find out why, and if you were OK.
A totally platonic mate would ask, someone you only know in passing might even ask, but someone you are attempting to forge a relationship not asking? ALARM BELLS!
Your inital feeling was to end it. You have now talked yourself round and will then be endorsing his lack of care toward you, this will happen again and again.
If he were into you, he'd go public, he'd make bloody sure you were ok and if not he'd ask what he could do.
To me you are selling yourself short, and that means you don't recognise your own value. Dangerous territory. If you don't treat yourself right, look out for yourself and lay down boundaries, codes of respect, how on earth will others know how to treat you as they should.
don't compromise. He let you down, from here he's not even a friend, let alone a boyfriend - you have to see that?