This.
OP, there's literally nothing to be confused. Just because he says it's a two person thing etc, doesn't mean he will change himself for you. Do not expect him to become 'normal' for you, it's not going to happen. It should not either, he likes what he likes, that's his right. As it is for you, to like what you like.
There's just two options before you.
If you want to try his way of lifestyle because it intrigues you and it doesn't make you go ick no way eww, then you can consider meeting him this weekend, give him a chance, etc. Be aware that it will come with its own pluses and risks, just like any other lifestyle (incl monogamy).
If poly/swinging lifestyle is not for you (it def is not for everyone and with good reasons), if it makes you uneasy, if you are strictly into monogamy, then respect your own preferences and boundaries THEY ARE VALID no matter what he or anybody says, and just block him. No more contact or conversation.
Plenty of other fish in the sea, really, that do not tie you up knots and make you so worried/confused.