Op, if is was your pfb I would have completely understood, because I've been there.
My pfb didn't have any sweets or chocolate until she was 3.5. She is my one child who cannot control herself around sweet stuff.
The other two had sweets and chocolate earlier, and we never made a big deal about them. They were treats, to be had occasionally. And they still are treated as occasional food.
My 5 yo leaves ice cream and cakes when she is full. If she didn't know if another cake came her way she would be scoffing the lot, not listening to her body.
We buy chocolate every week, a bar for each of us. Sweets I have stopped buying because my ds has braces and cannot have them.
Tbh I was far more annoyed at a party recently when there was only high carb processed junk food and squash on offer. Not even an alibi bowl of carrots. THAT is unhealthy, because that was supposed to be the main meal.
There is nothing wrong with having your children's healthy diets as a priority, but at a party it is very normal to have sweets, chocolate, cake, crisps, etc.