Who said anything about flabby and balding?
You said 'dad bod' which does imply a certain lack of physical tone. And most men in their 40s and above are well past their physical prime.
The fact is that men who find younger women attractive are far more likely to have the feeling returned than the other way around.
A well-maintained woman in her 40s or 50s has a much better chance of a fling with a fit young man than the reverse. The internet is heaving with middle aged men looking to get with young women. The vast majoity only get laughed at.
And not only by desperate and unattractive young women-there are plenty who are very attractive and like an older man.
Depends what you mean by 'older'. 5 years? Sure. 10 years? Maybe.
But 20 years? Not so much. Look at the stats. Most marriages are between men and women close in age. Big age gaps are strongly correlated with divorce rates. There isn't a huge population of attractive young women out there desperate to hook up with divorced dads in their 40s.
It seems some women respond to that fact with angry jibes trying to dismiss any older man as a physically gross old perv. Simply not true however much you don’t like it.
There are some fit older men. And older women too. However, attractive young people tend to prefer other attractive young people.