going against the grain but depending on the layout of your house I can see why you'd be annoyed, if he literally had to, for example, open a gate and go through into your back garden then that's clearly your private property and not appropriate for anyone to randomly access without permission - for deliveries, gas inspectors, anyone! Yeah you were just cooking this time but if you'd been sunbathing in your back garden or shagging in the kitchen or getting changed or crying for whatever reason, (all of which its reasonable to expect privacy in your own home!) you don't expect someone to be there!
I've actually had a similar thing from amazon, they somehow got over my back gate (which has 2 locks and is nearly 7 feet high!) and then into my shed in order to put a parcel there, whereas it would have been just as safe behind the bin in the side access (where literally every single other one of my amazon parcels have been left!) - it made me feel really uncomfortable to know that someone had effectively broken into my garden while I was out and I was worried that they were taking the opportunity to see how easy it would be to break into the house. I actually tried to contact amazon to ask how on earth they got in and why but couldn't get any sense out of them.
For those saying it's okay, okay it's daytime at the moment, but in 2 months it'll be dark by 4pm - would you really be happy if you were alone in your house (particularly if you lived alone) and suddenly looked up to see a random man staring in your back window? Because it would scare the crap out of me!
However if there is no real cut off between your front/back so he could, for example have just been going round the side of the house to check if there was a shed or a bin he could hide the parcel behind (which I think they're supposed to do if possible rather than just leaving it propped against the front door for anyone to nick, after all HE didn't know it was just toothpaste!) then that's fair enough of him.
I think you've been too honest in your OP so people are jumping on the bandwagon to slate you for not answering the door - if you'd said you didn't hear it or were in the shower and then saw someone standing in your back garden I bet you'd have got different responses!