My experience -
We began downsizing in our early forties when only DS went to uni (moved from a 3500 square ft, six bed house in town to a 2500 square ft, five bed village house with 0.3 acre garden).
The village was actually noisier as it was on an A road with lots of rumbling agricultural traffic and speeding cars - not helped by being on a hill that motorists loved to ramp up speed as they approached. Our three storey Georgian house frequently shook! Lots of other rural noise that we didn't mind but that was definitely more apparent than what we'd been used to in town. I had huge regrets about selling our old home and buying this one, although the garden was amazing. DH who's a bit deaf, didn't notice or care!
After a few years we decided to go more rural and bought a neglected detached mill with 0.5 acre in a very rural part of Wales (moving from south of England). So we downsized again (1300 sq ft, two beds) and thought that was it - no really close neighbous, a project, plus a wilderness requiring taming into a garden. We didn't realise there was a big house (10,000 square ft, huge grounds) in fairly close proximity that occasionally held weddings which were quite noisy and as a previous poster said, there was lots of motorbike/boys racer noise at weekends. The rest of the time it was too quiet!
Last year (now mid fifties) we sold the renovated mill with stunning garden and returned to England, albeit to a smaller town. We've swapped our 0.5 acre for a large courtyard garden and our two beds for three (similar size cottage to the mill). It's another renovation project although DH is inundated with work here, so barely has time for ours! We're also having fun creating another garden - just on a much smaller scale - and are finding it challenging, yet more rewarding than gardening in Wales was (a lot of what we initially planted there didn't take due to the wet conditions, so we eventually spent thousands getting it right). We've already crammed in hundreds of new plants and even went to Chelsea for the first time for inspiration.
Yes, there's different kinds of noise here (lots of seagulls as we're two minutes walk from the seafront!) and obviously more neighbours who all seem to be doing noisy building work - but so are we! We did make sure we bought somewhere with no overlooking neighbours so our garden feels extremely private which makes a big difference. It's also in a conservation area that seems to attract mostly retired/middle aged residents, so no screeching kids!
Overall it's been easier to (re)adapt than I'd expected. We're also far closer to family - no more ten hour round trips, thank god - and there's so much more to do/places to visit here. We love it and are so pleased we made the move.