My DH was a victim of domestic violence. It didn't affect us in the way it sounds like it's affecting your relationship. We were still able to go out places etc.
We've been together 2.5 years now and some of the damage still crops up now and again. You just need to be patient and understand that some of the ways they react is not a reflection on you.
For example, for a while it felt like whenever he made a statement and I agreed he'd continue to drive his point, even after I'd agreed, like he was expecting me to disagree with him. Like he had already lined himself to have to prove a point or give extensive evidence as to why he was saying what he was saying. I'd get annoyed sometimes because he would keep going on and on trying to convince me of something I'd agreed with 10 minutes ago. Sometimes I'd say "ok" and he'd get annoyed because he thought I was saying ok just to dismiss him rather than because I was agreeing with him, I was ok with what he was saying.
I remember he was 5 minutes late for lunch at my house once and he phoned me in a panic, apologising profusely and explaining his reasons in great detail.
If we ever argued he'd say "why are you even with me if I'm so bad". This could be over something really silly. I recognised that this was him testing the water so see if I would leave him. He used to ask why I was with him and made comments about me being too good for him. He's a wonderful man, excellent father, he just didn't see it as his self-esteem had been knocked out of him. It took some time to build that back up, but he did the same for me.
Any time something cropped up, after the dust had settled we talked about it. He knew I was nothing like his ex and I wasn't going to flip my lid at him being 5 minutes late but he was with her for 12 years and lived like this for 12 years so some habits were hard to drop.
He was very open with me from the start. We'd known each other 20 years so I wasn't exactly a stranger.
We said "I love you" quite early on and we even talked about getting married after a few months. Some would see it as a red flag but the way my DH explained it was that he'd spent 12 years reading his ex's behaviour, language etc to try and anticipate incoming explosions of rage and obviously to try and avoid them. He admitted that he was closely analysing me at first and it didn't take long to work out that I was genuine. He felt safe with me and able to open up without judgement. I had some damage too from previous relationships so it probably helped that I was sharing my vulnerability with him as well.
I've met his ex because he co-parents with her. I've had her screaming abuse at me at my door, she's even bragged about hitting him and how she would do it again. I can see how anxious he still gets if he ever needs to contact her for anything regarding the kids. We've both put boundaries in place to minimise contact with her.
Everyone is different. If you're serious about getting into a relationship with this man I'd suggest he gets some therapy first. If he's still closed off to you and won't take you out, he definitely has issues he needs to work through before even contemplating a relationship.
Time, patience, understanding and communication is the key. But don't lose yourself. A relationship is a 2 way street and you need to be getting something positive out of it as well. You're not a therapist or a fixer. Yes, you can help but he really needs to get to a stage where that's not all you're doing and you can both get some enjoyment from the relationship.