You will probably find that there is a last minute rush, Don't cancel the party just send a brief reminder out.
When my dd had her party some people didn't reply until the Thursday before the party, which really pissed me off. I don't think people mean to be rude, it just slips their minds.