Love is the key. Loving each other enough to want to give each other pleasure and the closeness that physical intimacy brings. It may mean finding different ways to accomplish that; a couple's sex life evolves just as everything else in their lives. Giving pleasure can be enjoyable and even hot for the person doing the giving especially when love is involved.
It sounds as though your husband is trying to bring you pleasure; Viagra, sex toys, and letting you know he fancies you. You even said he puts off his own pleasure to try to make things last longer for you. Sex isn't only intercourse; there are SO many other ways to have fun. Most couples will eventually go through this, but mutual love, respect and the willingness to be unselfish and try doing things differently are key to maintaining a satisfying relationship.
DH bought me some toys a while ago but I was a bit horrified - not because I'm a prude but the idea of doing things like that with him grosses me out a bit.
What are you doing for him? Besides hiding the toys, avoiding the issue, rejecting him, making him feel you find him disgusting. And not being kind enough to schedule things 30 minutes in advance. All the while you're daydreaming of sex with other men. If you don't think he feels that, if you don't think you're hurting him, you need to get a grip.
It feels a little like you're looking for approval to cheat or wreck what's left of his self-esteem by asking for an open marriage. You've unilaterally removed sex from your marriage. Not because he's abusive, or unkind, or a cheater. Basically because you've always had difficulty maintaining long-term sexual interest.
I'm sorry, but I think you're being incredibly hurtful and selfish.