By observer I don't mean that I don't join in, more that I always feel slightly detached, and find myself analysing a group, even if it's my own family.
I can talk the talk in that, if I'm around the posh side of my family, I can speak the way they do and find things to talk about, but I've never been skiing and I don't go to the opera or the other sorts of things they grew up with. They tend to assume we have a shared frame of reference for these experiences, because they don't know much about what my childhood was like (for various complicated reasons my dad, their uncle, was estranged from the family, so I've only known my cousins as adults, though we are friends now).
On the other side, my mum's family is very working class, and again I can join in and chat about stuff when I'm with them, but I didn't have that childhood either. I'm not one of the gang.
The only time I've ever felt like I really, truly belonged was with the friendship group I had aged from around 17 to 23. I think it was a product of the very specific social mix and environment of my school, which allowed us to find each other. It naturally broke up when people moved away and became adults, but I'm glad I had it.
I think music is can be very difficult to get into as an adult, I find I'm rarely interested in anything new, even in the genres I like. I think for a lot of people their music taste is fixed in adolescence and that's that. Music though is not often a sticking point for universal experiences—loads of people don't like the Beatles. Even among peer groups people will have wildly different taste.
I agree TV tended to be more universal, but our generation (I'm early 40s) is probably the last one for which that is true, because we grew up with four channels. There's so much to watch now that you can't have those "did you see?!" moments anymore.