"quite a few rich older men with young wives...."
Sorry to read of your situation, and think your husband is being dumb. It does often seem the case that a rich older guy has "something to offer" but (without wishing to offend anyone here with an older partner) there may sometimes be the hint of a "gold digger" when it does...
I won't go into my (limited) experience on dating sites, but sufficient to say that with minimal information a bunch of young women claim an interest in meeting me without ever seeing a photo. If I filled in much more detail, there may be some interest from more mature women, for me as a person, not as a cash machine (if I put down that I'd been bankrupt, do you think it would dissuade any twenty-somethings, or do you think they're too air-headed to know the word?)
Yes, lots of men have fantasies about someone in the age group for an 18-30 holiday, perhaps even think they'd be able to take on 2 girls that age, but it's bound to be wishful thinking for most. Of course, some might use their position at work to flatter some young thing, but I suspect many young women would curl up their nose and tell a girl friend how much it'd make their skin crawl for someone older than their own father to be making advances.
In your 30s, you're young and could start a fresh relationship, but love and loyalty probably prevent it, however much of an ass he is, because family stability is high on the agenda, I'm guessing.
You don't need his kind of tell the truth revelation - it seems so hurtful for him to have suggested he could be thinking of someone else.
Unfortunately, if I made some comment about 'enjoy it when heads turn' I might get some backlash from the feminist brigade, but you perhaps need some positive comments from a man or men, to reassure you that it's only boring older hubby who is negative.
Seems he is burning any bridge to future happiness by his insensitive comments, and yet you are far from being on the shelf - cannot think that making him jealous would do anything positive, but a heart-to-heart (in the tell the truth spirit) saying how he has upset you and damaged your confidence and love of him, just might sink in.
Cannot add much - keep your chin up - and good luck.