yes, I hate it too. I spend ages going through google maps and trying to work out junctions before hand. It's particularly city centres, onw-way systems, places where the lanes don't work as expected - you can be looking at so many signs, and it's easy to miss one. Spiral roundabouts are especially bad, as the paint is often worn down, so it's really not very clear which lanes go where. in fact that is a problem in many places - paint is worn off, and the signs were too far back, or drivers covering the paint, and you can suddenly lose confidence that you are in the right bit. Or local knowledge means that people do unexpected things, or know that certain lanes are the ones to be in as others turn very quickly into bus lanes or something.
A lot of junctions look weird to me, who didn't grow up here, and I might drive past something and think 'no that can't be the turning' and then maybe it was, some little narrow road that looks line an alley, or some complicated angle that you think must be for cars coming the other way or whatever. Then you've missed it and you're stuck. The signs aren't always easy to spot and when you're under pressure with traffic everywhere else, frustrated if you are slow about it all, it's easy to panic. Mostly old town centres for this kind of thing, not set up for cars initially, so it's all a bit unpredictable how it's been dealt with. Bus lanes, timed bollards, pedestrian only bits at certain times or days, narrow lanes that you might think are one way but aren't, or vice versa. You can plan it out on google first, but it can still be confusing in person, or with all the extra traffic and roadworks and stuff. Or when you get there, you realise the route that seemed ok on google was not the best for some reason, and then have to navigate totally on your own.
There aren't always places to pull over - that's the problem I usually have! In town centres, nowhere to stop even, so you just have to kind of keep going into whatever complicated one-way system you've got involved in, trying to figure out how to get out of it into a residential bit where you might be allowed to stop.
If drivers were more patient with others, it might be easier, but most people are in a hurry to get where they need to be, and have little patience with people who seem unsure about anything.