*I am useless at confrontation too but I still think I would ask, pleasantly, the non-confrontational question: “When I told you I needed to lock the gates, why did you ask me to give you 5 minutes?”
And then just… silence. Resist the urge to offer answers for him. Make him explain his thought process.
If needs be, you can always follow up with: “I came back 5 minutes later, as promised. I didn’t see you scouting the road for other parking possibilities. Did you even try to park elsewhere?” And again, wait.
If feeling daring, you could add, still pleasantly, and as if you are simply bemused and want to understand: “Was it because I’m a woman that you thought it wasn’t important when I told you the gates needed to be locked?”*
These are good. I would also recommend a kindly maternal, "Well, you have been a silly boy, haven't you?"