Right. This is important, OP, and it will stop you getting into any more abusive relationships:
Any time when you find yourself saying 'Is it just me, or...?' about your partner, IT DOESN'T MATTER IF IT IS JUST YOU. If you think you might be being paranoid, IT DOESN'T MATTER IF YOU'RE BEING PARANOID. If someone says you're oversensitive, IT DOESN'T MATTER IF YOU'RE OVERSENSITIVE.
I'm sorry for the capitals, but it's SO important. If you're feeling paranoid, it's because he's done something that makes you uncomfortable. You can question why you feel uncomfortable all you like ('is it me/is it him/am I being silly?' etc), but the raw fact is, he has done something that makes you feel uncomfortable. That's the only thing you need to pay attention to.
I have a friend who cannot be touched, however innocently, on a certain part of her body (similar to being touched on the knee) because it was the precursor, when she was a child, to sexual abuse. She dated someone who persisted in touching her in this way, telling her she was being 'silly' to be so sensitive about it, and that it was just an innocent display of affection. For anybody else, it would have been. But she had a boundary there, and whilst her date wasn't doing anything 'wrong' in the eyes of the world, her clearly stated boundary was being disrespected, so she ended the relationship.
That's how it works. It doesn't matter where your boundaries are. Nobody can tell you they're wrong. They are YOURS. Your feelings are what make you YOU. They are what differentiate you from everybody else. You have to respect them. You HAVE to, because otherwise you end up mixing with people who make you feel unpleasant. Like the guy you're posting about.
This is how it works, in all relationships (friends/family/colleagues/partners/everybody): The person does the thing you don't like. Then you calmly and clearly tell them that you didn't like it. Then you judge where to take the relationship, based on their response. If they are sorry they've upset you, and want to reach a compromise so that it doesn't happen again, good. Spend more time with them. If they defend themselves, make excuses, or try to make you feel bad or guilty for your feelings, then distance yourself from them.
That's it. That's all you have to do. That's everything you need to know about 'developing healthy boundaries'. Lots of people have trouble recognising their feelings to start with, but you're already aware that you're uncomfortable with this guy, so (strangely) that's good, because you're halfway there.
You feel 'pent up' because the feelings you're keeping quiet are the real you. The big, strong, self confident you that says 'Hey, fuck you, mate!' to people who try to walk over you. You are squashing her down. Get a pad and a pen (a red pen maybe) and write down what she's telling you. She may wish to use capitals for a while and quite a lot of exclamation marks, but let her speak, just to you, for now. I expect that she will swear and be in a very ill temper. Write down the things that are pissing her off most, to start with. I will bet you £100 that once she starts you'll have trouble getting her to stop. Welcome her, and all her anger. She is the real you, and once she gets it all off her chest, she's going to lead you into a healthy, non-abusive, self respecting life.
You have to listen intently to every word she says. She is enormously important: she is you.