DD1's school is mixed in this respect, I think.
I am a Governor there, so I can be in the school for a reason, other than DD1. Having said that, I will always make an effort to be discreet, because much as it is nice for me, it is disruptive for DD1 if she sees me. It is her school, not Mummy's. So, if I am in at particular times, I will walk the opposite route to the one I anticipate she would take. It has been known for a member of staff to pull me into a room because DD1's class is suddenly appearing at the top of the corridor!
The Head Teacher has an 'Open Door Policy'. That means that you can turn up at the school to discuss a concern at any time. If she is busy, you have to wait, but she will try and fit you in. Or you can phone. Any queries for teaching staff/SALT staff (there are 3 based at DD1's school, although they are NHS SALTs and serve other schools as well) can be made by phone or a note in the home/school book.
There are fortnightly coffee mornings, and we are encouraged to invite anyone who may benefit. So at present, we have 2 parents who have children with SN at Mainstream schools, but feel isolated, so they come to our coffee mornings for the SN support.
Parents are very much encouraged to be in partnership with the school.
HOWEVER, and this is where I think the answer to your AIBU? is Yes:
SN schools are increasingly shifting in their profiles. Survival of prem babies is going up, the number of children with severe and complex SN is going up (not just as a result of prematurity, I hasten to add) and the result is an increasing severity of the profile of children in Special Schools.
Even 'MLD' are tending to get more 'SLD' and 'PMLD' children, and the 'MLD' children of a few years ago are being pushed out into Mainstream education.
The school staff have to balance an incredibly complicated set of children, and their needs. They also have to house the teaching assistants, any equipment, professionals (SALTs, OTs, Physios, Ed Psychs, etc.) In DD1's class, a more able class, there are 10 children and 6 teaching staff in one classroom, before anyone comes in to see them. Other more profound classes will have standers, kaye walkers, achieva beds, supportive chairs, etc.
Imagine also, given DD1's class, if all 10 parents decided to 'pop in' for a few minutes. You could have an hour and 40 minutes of teaching time gone, with each parent just 'popping in' for 10 minutes. Add to that, the amount of time it takes to refocus the children, who often have limited attention spans anyway. Additionally, the staff, with the best will in the world, will be diverting some of their attention away from the children to the parents.
Then, you have the issue that the parents don't know the profile and the needs of the children in the class. They won't know if behaving in a particular way would upset 'Charlie'. An example, is that if I went in to DD1's Yr R class for any reason, they couldn't do anything with one of DD1's classmates, because he was so fixated on my mobile phone (that just happened to be in a holster clipped onto my belt loop) that he would completely zone out anything else.
It is hard to know that your child is away for 6 hours, and you don't hear anything from them at the end of the day, and desperately try and glean the tiniest scraps of information from them. But, school is their 'work' time and it is for them, not us.