How old is your DS? Would he be starting in year 4? Or in their new(ish) pre-prep?
The first point is that summer fields is a prep school - would you be looking at boarding or day for year 9? If day, is there a suitable school in Oxford that you have your eye on?
If boarding, do the destination schools line up with what you want? E.g. basically Eton, Harrow, Radley, Oundle, Winchester etc.
Are your younger children all boys? If you have a daughter or daughters, where would she be educated?
Sometimes Wetherby pre-prep sends boys boarding to summer fields - can you be but in touch with former Wetherby parents to see their opinions on Summer Fields?
Another thing is that Summer Fields is a predominantly boarding school that takes some day pupils (as opposed to a mostly day school with a boarding house or a truly 50/50 school). Your son will most likely be fine (perhaps better off!) as a day boy in year 4 and 5 but may be missing out and asking to board after that. If all his classmates are together 24/7 and he's not it maybe be trickier to make and keep friends. I know the school has a pretty generous weekend leave policy but I don't know if they have flexi-boarding?
A good thing about moving to Oxford is that even if the school doesn't work out there are a number of great options.
But there are amazing boys' (and mixed) preps in lots of locations. Have you considered some of the others?
Have you visited the school? It seems mad to apply without seeing it, but suppose it will be very common in covid times - and probably many international boarding families have chosen schools sight unseen!
What would concern me is:
a) can you both keep your jobs?
b) do you own property in London? If so don't give it up! Rent in Oxford and worst case if it's a total mess you can move back
c) do you actually like Oxford and would you like to live there as a family
I don't think you're silly to consider this - it could be a great decision for you and your family BUT it is a huge decision and you may be putting too much pressure on this school to be perfect - which may end in disappointment.