He wants the ability to see other people and have a sexual relationship with them without the label of being an adulterer. But he also wants to keep this lifestyle he has with you. Problem with this is it's all his way, no compromise on his behalf and you get half a life and will have to eventually watch him build a relationship with someone else. Then he'll drop you like a hot potato, and move in with his new interest and she'll be his best mate. You'll be left as a shell of your former self and feel even worse than you do now.
I know it's painful but please read this OP. This is a glimpse of your future if you agree to his weird and frankly arrogant as fuck suggestion.
He gets to have his cake and eat it.
You get the odd crumb and to watch him eat his with your face pressed up against the glass.
Fuck that. You'll end up a shell of the person you are and the person you could be.
You sound very submissive, passive and conflict avoidant. I would strongly suggest you have some counselling WITHOUT him to get your head around all this and also to develop some skills around setting appropriate boundaries and saying no.
Because he will leave you fully at some point and when he does, it'll be every person for themselves.
Put yourself first. This relationship is too broken to be fixed, but you aren't.