My DDs state school has varied between 120 and about 90 pupils in the time I've been associated with it.
My observation is that around 98 was the minimum that worked.
The problem is that although cross age group friendships do form children do like their main friends to be in their year group.
As others have said below a certain class size gender imbalances seem to occur and suddenly you'll have a class with only 3 or 4 of one sex.
Clearly any child that doesn't fit in that group is liable to leave and sadly other parents follow because the remaining DCs feel the lessons are biased to suit the majority.
We now have a single gender year in a mixed school which is far from ideal.
A small school with a balanced intake of 16+ a year is absolutely lovely. It may not offer the clubs and sports a larger one does, but the feeling of community compensates. Remember a school of 300 still only has on HT, but an awful lot more parents wanting to bend their ear.
DD1s enormous secondary works because they have a very clear pastoral care system and brilliant admin staff. I doubt all, but the largest primaries could afford this.