By sweets I’m assuming you mean like packaged candy, rather than using the word to mean like any sweet food?
I don’t think it’s unrealistic to do it until the age of 5, especially if born early in the school year. My DDs never had sweets at that age - they weren’t “banned”, we just didn’t buy them and they didn’t know they existed in order to ask. It wasn’t even a decision I made. I don’t eat sweets so just never bought them.
Lots of parents don’t allow all sorts of things when their children are under 5. I doubt many 4 year olds drink coke, but that doesn’t mean that not allowing them to drink it at that age will result in a cola addiction later on.
Once they start going to parties and having party bags, then no, I wouldn’t be confiscating items because I didn’t allow them (although I don’t allow some that are choking hazards, but I say why and swap it for another sweet/chocolate thing that I get for them).
I still don’t buy sweets. I don’t really see why I would. We do a lot of baking though so I’m not anti-sugar. I just wouldn’t spend money on horrible sweets. My DC are yet to show any lack of control around the contents of a party bag or a school fete haribo mix.