Yes I second therapy. Could you get some counselling for yourself to build up your self-esteem?
Something is keeping you trapped in this cycle with him. You know it's not healthy. You know he's not nice. You know what he says about you isn't true.
So let's have a look at what you're getting out of it. Do you feel that this is all you deserve? It's not. You sound really nice and caring, you deserve to be cared about in that same way.
I'm sure I've told this on here before, but after divorcing xh, in the early days of dating my now dh, he offered to make me a sandwich and meet me at work to have lunch together (yes! This is a very boring and stupid story!)
So he turned up with the sandwich he'd made, and I looked at it and said, "that's my favourite", and he said, "that's why I made it".
I burst into tears 
More than a decade with xh and he never remembered my favourite anything. This man was not only listening to me, he wanted to give me what I wanted (not what he decided I wanted, but what I actually like).
I bet you know his favourite things. Does he know yours?
Because a relationship is about being with someone who you are compatible with. Someone who makes you feel good. Someone who you can be you with.
You deserve better. You deserve somebody who respects your decisions.
Right now you feel confused and lost. He has done this to you on purpose. It's not an accident. He's got you jumping every time he snaps his fingers and panicking about his reaction when he's not even there or actively doing anything.
What do you want?
Are you happy in your life? Work? Friends? Hobbies? How can we help you feel better? Because you need to get to a place where you're not being drained by this man. Where you're free to be you.
Counselling can really help with this. It's great to have somewhere to deal with your self-esteem.
I don't know if anyone's mentioned the "freedom programme"? Google any groups near you. They specialise in helping you see the negative patterns in relationships. But also look for personal counselling just for you.