What if it is really me and I’d be throwing it away for nothing
This is the flaw in your thinking, and what's making this feel like such a dilemma. The feeling that if it's 'really you', it can be regarded as 'nothing'.
Is IS you. The problem that you're having in your relationship is 100% down to you, because you are the one who is having the problem. But why do you think that that means you could potentially try to put that problem aside, or ignore it? Why do you want to minimise something, as if it's just 'silly old me and my silly insignificant feelings'?
You are not happy. Your boundaries are being crossed. Something in you is telling you you've had enough. See? It's all about you, and it's your responsibility to make you happy, by putting yourself into situation and with people who make you feel fulfilled, safe, loved etc.
Reconcile yourself to the fact that it doesn't matter who is right or wrong, here. You have to take responsibility for yourself, and remove yourself from situations that make you unhappy. No blame, no shame. There are no rules regarding emotions anyway. He can do whatever he wants, and you can feel however you want in response.
So yes, this is 'really you': the real you, the core you, the you that you need to look after in order to be happy. If you can stop doubting that core you, this will be so much easier for you.
You know you can't stay and be happy, so validate those feelings, that thought. Validate that it's ok for you to do what you need to make your life the way you want it. He doesn't meet enough of your needs. That's all you need to know.