I'm on your side of the debate, OP but I do think YABU.
My DD never had any sweets or chocolate until she was 3, going on 4. Like Tokengirl, I just said that I didn't see the point until she knew what sweets were and asked for them. I had also read some research that said the food a child is given up to 3 will set the tone for their tastes for the rest of their lives. I remember saying something on Mumsnet at the time about my daughter not eating sweets, and was told too that she would end up obsessed with sweets, sneaking them behind my back etc etc...
Well, she's almost 11 now, and I do feel vindicated because whilst she doesn't mind sweets, she can take them or leave them. She'd still rather eat a bag of carrots! We did have a but if an issue when she was 5/6 and she was swapping out her sweet thing (cake, chocolate coin etc) in her lunch box with other children so she could have their carrots or cucumber instead, but that was the extent of the issue. We had to stop giving her sweet stuff because she was always giving it away. She actively dislikes fizzy drinks too.
So I get where you're coming from, but at the same time, I don't think you're right to dictate what other people do at their parties. As someone else said, at 3 I just used to swap the haribo for one of DDs treats. (She was veggie at the time, so we didn't do Haribo, she still doesn't, but she's not veggie any more - her choice). If they've never had it, they don't know what it is, so easier just to swap out.
When going to parties, keep a supply of your child's favourite treats with you, and when they get given some sweets, then swap them over.
That said, as they get older you do have to release the reins somewhat, as you can't have the same control over what a six year old eats as you can a two year old. It's a tricky balance to get right though!