You need to start respecting your feelings. You feel scared, but you're not sure if you should. But who makes the rules about what you 'should' feel? Who decides what's 'normal'?
I'll give you an example. I've told this story on here before: my friend got a new boyfriend. He was lovely, and she really liked him. In the evenings when they were watching tv, he liked to gently stroke her arm. A completely normal and sweet thing to do. But her childhood abuser used to do the same thing prior to 'taking her upstairs', so it triggered horrible unsettled feelings in her. She asked him to stop (she didn't explain why), and he wouldn't. He said that, because it was a totally normal thing to do, he shouldn't have to stop doing what came naturally to him. So she left him. Not because he was doing something unkind or 'not normal', but because he didn't respect her clearly stated boundary. She's been with her subsequent partner for years now; they're very happy and have a healthy relationship.
Whether your partner's behaviour is 'wrong' or 'normal' isn't the issue. You don't have to put up with anything you don't like, just because it's 'normal', so whether you've forgotten what 'normal' is isn't relevant here. You may have anxiety issues from your past, but that doesn't mean that you just have to put up with feeling anxious, now. It means that you and the people close to you might have to work a little bit harder on maintaining calm. Anybody close to you who will not support you in this needs to not be close to you. In the same way that it's understandable for someone who has been bitten by a dog to be afraid of dogs, even if they're friendly, you are allowed to be afraid when someone feels intimidating to you, even if they're not being openly mean or scary.
All of this is written disregarding the fact that he has scared you on purpose, because everything above applies to you and all your relationships. It's not about him. But really, neither is this: create as much distance as you can between you and anybody who deliberately tries to make you feel uncomfortable, in any way, including feeling scared. Just get away from him, and anybody else who does that.