We've owned larger houses with small gardens (the biggest discrepancy was a 3500 sq ft, 6 bed house with a 65 x 35' garden) and currently own a smaller house (1400 sq ft, 2 beds) with a 0.5 acre garden.
Gardens are very important to us, but personally I actually miss the larger house (2000 sq ft, 4 beds) with a 80 x 50' garden we sold to move here five years ago.
At the time we both believed a larger garden was more important than house size and we particularly wanted somewhere we could create a large garden from scratch. As we don't have DC living at home, it was more about an interesting outside space with loads of plants than a huge lawn for ball games/trampoline etc.
Five years on from buying somewhere that we thought had sufficient inside space (it doesn't) and a completely blank canvas garden-wise where the previous owners hadn't bothered to cultivate much except weeds, we've barely planted up two thirds and despite me previously absolutely loving gardening, I'm now finding it totally overwhelming. In part that's because of the challenging weather where we are - we've lost thousands of £££ worth of plants which is deeply disappointing and soul destroying. I appreciate it's a first world problem and we're extremely lucky to have what we do, but it makes me so sad to think of what we've spent hours working on, only to lose so much each year.
Part of the garden to our very old, rural property is a high-walled courtyard leading off to one side of the house. When we first bought the house it was a bare, ugly decked area with no plants whatsoever, but we transformed it into a little oasis with veg beds and lots of pretty climbers as well as somewhere that's roofed for an outside sofa. It's about 25 x 15' and I now actually spend more of my time there than in the 'main' garden. It's very private, not only because of the walls but because it's gated so it's safe for our DDog too. When I'm out there I try to pretend the rest of our garden doesn't exist.
It has made me realise that when we move (looking to sell soon, hurrah!), I'd be happy with a considerably smaller garden as long as the house is a bit bigger.