The key question (IMO) on whether to give up any specific food is whether you can moderate it or not, if it’s a food which is not wise to consume in unlimited quantities.
If you CAN - then no problem, include it as part of your overall eating plan. So, for example, I can, and do, moderate alcohol. I seldom drink, and when I do, I usually enjoy it and then stop after one or two glasses, and then simply don’t think about it for weeks or months. But chocolate, crisps etc - if I have some, I want more. And I’m either going to give in and have more - and the amount will escalate over time - or I’m going to have to fight myself really hard not to give in.
I would rather not do either of those. The only way for me to achieve that, from experience is to abstain completely. That gives me peace of mind.
Don’t get me wrong. I would absolutely LOVE to have a packet of crisps, or a pudding, a few times a week. Couldn’t hurt, right? Not on the weight loss front at least. But in my experience it WOULD hurt me. Either it would trigger more eating, or I would have to use huge amounts of energy not to overeat.
So at the end of the day, it’s not worth it to me.
The only other thing I’ll say is that I spent years rationalising that a little wouldn’t hurt, even though I could never eat just a little. I was unwilling to see the truth, until I was ready.