OP, you are getting silly answers on here because you have phrased the question wrong (based on the further information you have given).
It's not about choosing a partner over your kids - of course nobody should do that.
But your dilemma is in fact whether you should allow your adult children to control you and your life choices.
And of course the answer is no.
If what you have said about the situation is true - and we have to assume it is - then your children don't have any reasonable justification for demanding that you end your relationship.
Touching somebody's shoulders isn't abuse. Telling an adult off is rarely acceptable, but as a one off incident it isn't abuse - it's a mistake many people make when angry or upset, and as long as they apologise and do their best not to repeat it it's perfectly forgiveable.
Does this incident, alone, give your adult children the right to demand you end your relationship? Of course not.
And therefore you would be doing your children, as well as yourself, a huge disservice if you gave in to their bullying, controlling behaviour.
You should fight for your relationship with your children, obviously. Do everything in your power to understand where this attempt to control you is coming from, and to help them see it for what it is, and to work on whatever underlying problems are leading them to act this way. (Because it's not normal. I would never, ever attempt to control my mother's life choices like this - and I would be beyond disturbed if my daughter attempted to do this to me).
But giving in to them would teach them a terrible lesson - and would also do yourself a huge injury. You are an adult - you can date who you want - your kids have no right to dictate what you do in your personal life.
My advice would be to meet with them both individually and have it out. Try to get to the bottom of what is causing them to behave in this inexplicable and cruel way. If this is new behaviour from them - if they have shown no signs of attempting to control and abuse you before - then try to understand what has happened to cause this drastic and horrible change.
It may be that on doing so you will uncover that there is more to the story - that your partner has in fact hurt them in ways you are not currently aware of. If that's the case, then obviously you should behave accordingly and do whatever it takes to get him out of all of your lives.
But that's a big if. If instead it turns out that the story is, on fact, simply what you have told us - then it's inexplicable, and I wish you luck in resolving this horrible dilemma and getting to the heart of what is causing your children to behave like this l, and hopefully resolving it. X