He doesn't want to sleep with you again because he is a bit freaked out by the intensity of your feelings. He knows he's opened Pandora's Box and that you are not being particularly rational at the moment and it's scaring him.
Perhaps ending up in bed together happened accidentally and gradually, or perhaps he is one of those men who enjoyed the game of reeling you in, getting you to a point where you feel emotionally dependent on him, and then dropping you once it was no longer a challenge. In which case he is the worst kind of bastard to take advantage of you while you were obviously feeling very vulnerable and mixed up.
Either way, he is either spooked or bored now, and no good can come of you trying to cling on to this brief dalliance/friendship. You must not start convincing yourself that it is a real relationship that can solve your problems or give you the excuse you were looking for to run away from your marriage - it really isn't. You are assigning him a pivotal role in where you go from here, but actually he is nothing more than a symptom, not the cause and certainly not the cure. At most, he is nothing more than a catalyst. It is very, very important that you understand and recognise this.
Don't make the mistake of thinking that the only two options are 1) being deliriously happy with him or 2) staying in an unsatisfactory marriage as a consolation prize because he has rejected you.
There is a third option and that is to stop being passive. You need to take control, admit that you are unhappy in your marriage and do something to change it. That can either be to leave, or to acknowledge out loud to your husband that you both need to work intensively on changing the things that are not right. And after that, you may still need to leave. Or you may not. I might all be fine. But you need to take responsibility for your own happiness and formulate a plan that does not involve the other man. Because believe me, he does not intend to become anything more than a walk-on/walk-off extra in this little drama he's dabbled in.
What you are doing at the moment is placing the onus on him to change your situation for you. You want to be able to say to your DH 'I love him and I need to leave you to be with him. It's not my fault. I can't help it. ' Or you are perhaps hoping that your DH to find out and make the decision for you.
Except that the OM doesn't want you, so now you are feeling trapped by your own cowardice, because leaving to be alone is too scary.
But it is perfectly possible and indeed allowed for you you to just leave your husband for no specific reason/person whatsoever. It is allowed, if that is what you need to do.