I agree cancellation is a bad idea, because the repercussions socially could be a bugger. It's miserable to have to risk your money because they are rude, but I think it's a lesser cost than the potential social one.
We had whole class parties when DS was a preschooler, and people barely ever RSVP'd but we always ended up with more than we asked - by about 20% - from siblings being dragged along. People are just bloody rude, in my experience.
I have had parties where a parent turned up after no rsvp, dumped her child and siblings ( no sibling was invited and I had no idea even what sibling was called) without saying hello to me or giving me any way to get hold of her. I hear her say to her Dd "eat as much as you can as I'm not cooking tonight" as I turned around she left.
Yep. Best of all was the parent who collared another on the way out, asking them to tell me that their kid had, "some allergies, quite serious". No clue what they were, nor which child was referred to! Fun. 
Most heart-breaking was the small party - 5 other kids - once he was at "big school", where all the other parents said yes, and only 1 child turned up. DS is autistic, though very high-functioning in the interpersonal sense if he knows people. But one mother messaged me that morning with a long sob story about how she had been called into work and couldn't get out of it and her CM couldn't get the kid to mine... and then next day she was in the background of some pics a friend took at a local farm park, with kids, at the time of the party. The others just never showed up.
We had a meal out plus activity for his next birthday, where we made the attraction so enticing we knew the kids would be desperate to go, and we made it directly after-school so the parents would have to see us immediately beforehand if they flaked. We wanted to soothe DS' hurt by letting him have a party that was an undeniable success. And this year coming, we're scaffolding him into the idea that he will get a really special trip to somewhere he wants to go instead, as "we can't afford both..." (true, but not the reason).
This stuff is an expensive source of misery, IMO. My toddler will have soft play parties only, where you need to chase for menu choices, and no extra kids can be thrust upon you that easily! (No problem, I should add, when parents asked beforehand about siblings - we all have childcare issues sometimes. It was the unexpected dumping - luckily I had some extra party bags and cake made up for family members!).