You mention a few important points. You want sex everyday, at least before your peri menopausal started. He is fine with 2 or 3 times a week. You currently have sex 4 or 5 times a week. You are picky and choosing, while he is not. He enjoys seeing you with many men. You have a nice body and can attract any man, but do not want to do so.
So tell me, with your current dampened sex appetite, you are sleeping together 4 or 5 times, more than what he normally desires. Does this mean you attract him to you to sleep more with you?
How many times a month you go swinging? Actually it is Group sex. And out of these how many times you just watched or loitered without interacting sexually with someone?
How is the quality of your sex? Does he satisfy you sexually?
His desire to see you with other men, may stem from his desire to wanting you to be satisfied sexually, if he is failing to satisfy you himself. This is one of the reasons why men fantasise or want to share their wives. But it is obvious that this is not your cup of tea.
I want to know when you swing (or Group sex), is he fully occupied by himself with another woman, or does he want you near him to watch you have sex with another person?
In your day to day life, is he picky how you dress, that is, does he want to see you attract men or women?
You say you have agreed to tell each other of attractions, and it is clear that he has had many over the years. Attraction is fine, as long as it is purely sexual. Have you ever discussed someone you were attracted to, or someone he is attracted to, in details? I mean does he elaborate on this. I am not sure if you intended to say that he meets with his attractions behind your back. Does he?
How old are your children? How is your recent everyday married life? Does it cruise normally like it did when you got married?
In your late 'forties you are still in your sexual prime. Women's sexual desires (and sexual aggressiveness) are highest between 35 and 45, with many till 50, if not beyond.