When I were a youngun, back in medieval London, going trick or treating involved mild levels of hazard. So a lot of people either didn't go, didn't want anything to do with it, or a few stayed in to do the "trick" part. You could choose to knock on a door and say "trick or treat" the householder could a) say "treat!" and offer you candy/chocolate b) say "trick" and throw water/fake blood/an egg at you or c) tell you to clear off/get outta here or I'll call your mum. Sometimes, if the householder chose b) or c) the trick or treaters would opt for "trick" and egg the house or similar annoying as f- behaviour. Perhaps I just grew up in a ghetto. It was the last century as well.
Now, it's lost all the halloweeny danger and is just a bunch of cute kids and often their polite charming parents in outfits expecting sweets. It's rubbish, just another reason for shops to pump out cheap plastic crap before they hit the main Xmas shopping fest. And if I dared to adopt the old school approach and throw a bucket of water or fake blood or slime or real blood at the supposed "trick" or treaters, they'd probably think it was reasonable to call the cops or something. And if they egged anyone's house, all hell would ensue.
So:
YANBU - If you're going to say "trick or treat" expect tricks. Otherwise just state that you're on a freeloading chocolate mission.
YABU - You are Halloween's equivalent of Scrooge. Wind your neck in and get the haribo ready