In case your position changes, I will offer my opinion! Our garden is about 100 feet x 40 feet. We moved into our house about 8 years ago, bought at the top of our budget at the time, and the garden was mature. Lovely. But we were not (at that time) gardeners, and basically we neglected it badly.
It meant that when we did become interested in our garden, and had time to give to it, about 18 months ago, everything was very overgrown and needed substantial cutting back. Most of what we had was very leggy and woody and it meant that we needed to start again with a lot of it.
Now we have a gardener come in for 3 hours a week, and has been doing this for about 18 months. Just over a year ago, we bought a load of shrubs from her, which she got from the wholesalers. Things like three syringa, fatsia japonica, euonymous, magnolia, acers, prunus, bay, laurel, lots of cistus, buddleia, hydrangeas, photinia, masses of lavender - loads of things. They didn't all flower enormously in their first year, but dear lord, they are now very established and are large and doing so well. I am now buying lots of climbers and have bought fruit trees and bushes. I have been surprised at just how quickly the garden has gone from being cleared, to looking really full. Things like mahonia, forsythia, hypericum, fatsia japonica etc will grow really quickly and time flies!
So, what I am trying to say is that you can quite easily create an established garden with a mix of structure, hard landscaping and planting. Even painting and cleaning up what you have. And you don't need a huge budget. If it were me, I would buy the bigger house, and invest in the garden when you can afford to. I am assuming that the garden is somewhere that still allows you somewhere to sit, for the children to play etc, so that it is useable.