"The reason I don't want to ask him about it is because he would get angry and stop speaking to me which creates an awful atmosphere."
So? If he get's angry then that's his own problem, also why would he get angry if he has nothing to hide? If he went off into a strop I'd tell him to grow the fuck up and that this conversation isn't going anywhere, whether he has a strop or not.
This is coming from someone who's DP has anger problems.
"I asked to look at his phone to which he said I could if I really wanted to stoop to that level."
I'd have replied with "Yes I will thanks, seeing as I know exactly what levels you will stoop too."
I would have still checked, does he have FB on his phone? Maybe you could access that through there too?
"think they use fb which has a password and he would never give me that."
WHAT? Really? I know all of my DP's passwords, if he refused to tell me his FB password given the circumstances then I would leave, again if he has nothing to hide then he will be open.
"He hasn't made any effort to regain my trust he exptects me to have gotten over it without any bother on his side. He has always given off an attitude of 'it's your problem' if I feel upset by anything he has said or done, he really cannot see that he needs to help me."
Well here's the thing, this entire situation is HIS problem. It's HIS problem that he betrayed his wife and had an EA and now it's HIS problem to fix if he wants to lead a happy family life.
You could leave it, you could get over him and move on with your life however he will always be left behind knowing what he lost and exactly why he lost it, this isn't your problem.
Remember something, you have the upper hand here, not him.
He is supposed to be grovelling at your feet for what he's done, he doesn't get to turn everything back on you!
His EA, his problem, his life he needs to fucking sort out.