Why? If you don't want people to come at that time, invite them for a later time 
I always think it's rather sad to have invited people to a party and then so many of them don't come when you ask them too, not because they are elsewhere but because they somehow think it's 'cool' to be late.
Indeed, it's rude, in my book.