While downsizing is great in theory, we get settled into our larger houses, with every closet filled with 'stuff'. Also by that time, there are grandchildren to look after and occasional sleepovers, large family dinners, a garden that you have watched grow for 20 years or more. And in a neighbourhood that you can afford!
That's just where we are. five bedroom house which is paid for, in a nice neighbourhood and a great view (which is important to us). We are in a smaller town, and anything appealing in a smaller size isn't really available in this town. So it would mean moving away from family, so most likely not going to happen.
We are early 60's, but I know in ten years it is a crucial time for having the strength and energy to downsize, pack up and go somewhere else. We have even purchased a two bed ground floor condo in a nice small city which is currently rented out waiting for us to make a decision.
My advice is to give yourself a few years after your mortgage is paid to travel, then purchase a smaller property in a 'retirement location', and rent it out until you are ready. Then it is partially paid for when you are ready.