OP, imagine you're running a business with another person, you both have 50/50 stake in the venture. Your lifestyle depends on the success of that business, if it fails you lose your home, your credit, your reputation. Everything.
Now imagine your business partner, whom you depend on to be there for you, to be invested emotionally in the business, to be present and accountable, to help bring in clients, to help close deals, to help bring in money so that you can pay your bills. Imagine that person told you everyday how disinterested they were in the business, how they just didn't feel like doing any work. Imagine they just kept asking you to leave them alone and stop bothering them. Imagine if they said
"It's not you, I'm just not interested in work at the moment, could you stop asking me to do stuff, you're stressing me out. I'm just not into this whole business thing right now"
How much confidence would you have in them as a partner?
Would you ignore what they were telling you and carry on as normal, or would you believe their words and conclude that the business was in serious trouble, you are going to lose everything because if they leave the business closes?
It might seem a somewhat crude comparison but you could look at your relation the same way. Men and women think differently, what is common sense to you, won't be to him. I suspect that he has put a wall up due to your distancing yourself from him. He will have thought about life without you, and once people start to think that way about their relationships things can deteriorate rapidly. You repeatedly asked him to stay away from you, you effectively told him you weren't interested in the business, and he, in response, began to consider going it alone. Protecting what little he would have left were you to quit.
How you come back from this is the question. You both have to rebuild the trust and connection which has died. You both need to get to a place where you trust the other person again, but real trust, not trust by numbers.
There's no point in thinking "well if he were a good partner he would have supported me and understood". Because he is thinking the same thing, that YOU didn't understand HIM and HIS needs.
Fixing this has to be something you actually want, not just something you're doing because you feel like you should.
Right or wrong, he is going to feel hurt by your constant rejection, for men, sex is a way of restoring the connection, for us, sex tends to be something we do after we feel connected.
Start slowly, but you have to be genuine,