"If they wanted to prevent people getting in they would"
My uni isn't a campus university. Some buildings are open access, but not all (like the medical school, for example, where you need ID to get through barriers, and the library). But this isn't really for the benefit of random people who might fancy listening to a lecture, but more for the convenience of students and promoting the idea that it is an entire university that you joined, not just one School or Department. Ditto for staff.
But if too many people abused it, we would have to tighten security.
But attending a lecture when you aren't registered for that course, or haven't asked the lecturer? In small lecture theatres, I would consider it as very bad manners -- and if asked, I'd expect you to leave. Because when I'm lecturing I'm not there for the general public, I'm there for my students. Okay, as long as only a few random people attend larger lectures, no one is going to mind too much. Unless, of course, they are running out of seats already.
But if it became more widespread, then more checks would have to be enforced. Being open works only as long as not too many people abuse it! And after all, you haven't paid any fees, some part of which pay for my time preparing and lecturing, and the upkeep of the lecture theatres!
For postgraduate lectures, I would certainly expect even student attendees (research students, for example, as opposed to taught masters students) to ask before sitting in. And we would tend to know, because at Masters level with overseas students, we often use lectures as a place to register attendance for visa purposes (as we are required to do in some way).
But in the main, many lectures are not going to be at a level where a non-student, or even a student from a different School, is going to be able to gain much -- you won't have the requisite background knowledge! Which is why most universities do have public lectures designed for a non-specialist audience.
Would you, after all, go into a cinema screening, or a theatre performance, that you had not bought a ticket for, even if you knew that no one was checking the tickets because they trusted people to be honest? It's kind of like that if you as a non-student use university facilities -- you haven't paid for them, whereas our students have!