OP - what is sad is that you are handing him all the power in your relationship on a silver platter. He may never willingly tell you the truth. So what, you will just spend your whole life in limbo?
You don't need PERMISSION or an ADMISSION to end the relationship. You know what you know. So take control of your own life and make the decision for yourself. See a solicitor. Serve divorce papers.
He knows that you know, but isn't willing to admit ANYTHING until he knows exactly how MUCH you know.
Text him saying you are sending his things to the address, as others suggested. Then simply ignore his calls. He will get angry. He will want to know HOW you know.
I would recommend that you actually give him a false proof at first and see how he explains it. You could say that a close friend of yours SAW him going into the address, so you researched it and it was her address.
At which point he will tell you that he went there to fix something, or he went there for a cup of tea, or whatever. And then he is like a rat in a bag, because he will have given you PROOF that he was there.
If he denies it completely then you just need to stand by your guns and say "well, I wonder who has more to gain through lying about this. My friend - who risks losing our friendship through telling the truth about what she saw, or you, who risks an acrimonious divorce through telling the truth? Hmm. Difficult....so difficult..."