to downsize?

We currently live in a large 3 bedroom townhouse, with a very large garden. It's in a really nice area, lots of parks and things, good commute distance to London. Schools are not great but we can afford to go private so not a big issue for us.
We used to live in Surrey - and I loved it there. And I don't think I realised how much I loved it until we left. I miss that it's always sunnier (I know it makes no sense but it's just sunnier there), I miss being able to take my dogs into pubs for lunch, I miss everything about it. In order to move back to Surrey, we'd have to massively downsize. The commute would be about the same.
We'd be looking at pretty much halving the square footage of our house and probably quartering our garden - even after increasing our mortgage. DH and DS are happy either way, but obviously the dogs will have a petition up by noon if they find out they're losing the big garden. There's nothing tangibly wrong with where we live - neighbours are great, local amenities etc...I'm just not happy.
What do you think? Does size matter?