Well, firstly, you haven't wasted 4 years. You have enjoyed those years and what you got from the situation, which was friendship and good sex, without commitment, which you felt you could not give - all good.
I would see it as "wasted" if, for example, you were both child free, he had promised you he would want kids in 4 years and then back tracked and left you with a ticking fertility clock.
Maybe this is good timing, as it's made you evaluate the relationship at a time when perhaps you can look for something more, as your kids are getting older. When they leave, you might want a partner to share your life with, go on holidays etc. You won't find that person when you're still focusing on your FWB.
If you think you have feelings for him, I'd lay your cards on the table and ask him if he wants more. If he does, that's all good, but if he doesn't, it's maybe time to let him go.
The age gap would bother me slightly, as you're really at different life stages.