I am in a similar position to you , OP.
Moved my children, 7 and 3, to a village school as had heard good reports and that the smaller class sizes made an enormous difference.
They did, the school is lovely and class sizes a positive influence on education.
However, this has been negated by the isolation both felt about being the only non-white children in school. Subtle and overt racism from other parents ( school was lovely when we did complain about few things and nipped them in the bud)
Regret my decision to have moved them in the first place. Previous school was bigger , white majority but more children to mix with and find your tribe, so to speak.
Positive ending: elder one hit into grammar school in London, loves it there. States the diversity makes everyone feel accepted and valued. I m now torn between looking into a bigger school for the younger one or leaving her where she is, hoping she will move into same school for secondary.
My suggestion to you: we don't know what is out there until we experience it. Go for it, have a back up in case the school doesn't work out. At 4, they are resilient and will cope with changing schools if need be.
Hope this helps.