Some things to think about...
I personally wouldn't care (lecturer) at all, I'm giving the lecture anyway.
However, in an anthropology class, which typically don't have the huge numbers like say first year law, you may stand out a mile.
I teach up to about 50 students, and this late in the term, if a new face came into the lecture, I would most likely approach them and ask why they hadn't attended seminars.
Your chances of being spotted depend very much on the subject and the size of the class.
Also, it isn't the same as a student from that university studying outside their course, at our uni any student can audit any course or access the online materials. This is because they have been accepted/paid.
I don't think it's immoral at all, but it's obvious that if lots of people did this then there would be a tightening of security about lectures (our uni campus is completely open to the public and encourages people to use the facilities).
I don't know where you are in the country, but as everyone says, many unis run a whole public lecture series, there's Cafe Scientifique, there's online courses for free (the MOOCS)- so not really sure why you need to get your fix by pretending to be a student. You could still access the campus, have lunch, wander about, and study these courses at home.
This may not be as thrilling to you though, as it seems to be more about the excitement of pretending, and less about genuinely getting interested in one subject (most lectures would require some background knowledge, or are in a series).