I don't know anything about affairs, but I do know about betrayal and losing trust and it takes something horrible away from you. Without being cheesy it always reminds me of that Emma Thompson line in "Love Actually" where she says if she stays she knows everything will always be a little bit worse.
Betrayals of all sort sully and soil what should be the key relationship in your life where you feel most safe, most assured, most trusting.
It's a very sad thing, it's a loss, and I completely understand the concept that he has taken away your ability to trust.
However, on reflection of my own situation, my husband who I absolutely adored abandoned our family when he became depressed and changed from the most wonderful man in the world to the most awful bastard you could ever imagine.
I could take the attitude that I will be scared my next man would abandon me (after all, my DH could not have been a more loving hubby so if he was capable of it anyone could be) but that's a silly attitude to take.
Millions of people get depressed and don't abandon their families. Millions of people stay married and don't have affairs. Some people act in ways other people would not dream of doing and that is the way of the world.
It sounds to me like your husbands affair has caused you to cash in your chips and leave the table for fear of losing again. You've given up. In that state of mind whether you are with him or not - you have let the affair destroy you.
It's not a case of "good or bad" people, but it's more about the choices people make. Some people might decide to have an affair while others would never get themselves into that position. Can you think back and unravel the parts of his personality which enabled him to behave this way? Were there issues between you that led to this? Did he have form for selfish behavior? Was he capable of deceit in other ways?
I certainly know in my case my husband had millions of signs over the years that he was the type to run away from problems. I chose to ignore that because I loved him, he loved me and we were happy. It never occurred to be that when he was no longer happy (and no one with severe depression is "happy") that he would just run off instead of standing true to his promises.
Maybe if you can unravel the causation or the signs it will make you feel safer with others. That said human behavior is unpredictable sometimes and you cannot protect yourself from getting shat on.
This has clearly damaged you and you are not free from it yet. You need to think about your husband carefully. Do you WANT to be married to him? Do you still love him? Do you want to get back what you had before? If you do, then yes, I think "where there is life there is hope" and you can work on getting the feeling of happiness back.
However, if you are staying out of fear or apathy then you are throwing away your life.