OP: YANBU to be upset that someone you didn't know just opened your front door. Maybe he saw lights on or saw you pop upstairs, maybe he was worried or maybe he comes from a knock-and-enter area, but even so it would be a poor judgement on his part.
I'm not so sure about the "phoning before calling round" though. I would say that if you're going to visit unexpectedly, be prepared for the visitee to be out/busy/masturbating, and politely go away again.
I'm bemused by the tone of some of the posts above though, the ones along the lines of "I thought everyone sealed their triple-deadlocked reinforced steel door before retreating into their panic room and only answering the phone if the caller's identity has been verified by the queen and the pope." I'm assuming it's basically a rural/city divide, with some family attitudes mixed in, and of course those who have had a bad experience would naturally be more cautious.
Our front door isn't locked unless we're going up to bed, or leaving the house unoccupied. Friends and family knock-and-enter, although I suspect my ex will be more hesitant since he and his wife were dropping DS1 off and wandered in whilst I was in the downstairs shower. "EEEK! Be with you in a moment. DS1, could you please close the bathroom door for Mummy and make everyone a cup of tea." I've never rinsed-and-dried so fast in my life.
I knock on doors for a living (I knock firmly, twice, then walk away). I'd never dream of trying the handle unless I was concerned for someone's welfare.
(If you're going to hide behind the sofa that's fine, but please don't let your toddler stick their fingers through the letterbox at me. It puts me in a very difficult position.)