Christ, how bloody rude! Her not you. Agree with greycircles re not giving reasons, there's no way I'd be justifying non attendance of a 4 year old's birthday party especially if they are pretty random 'friendships'.
I have drawn a line about parties, if it's a good friend, we go or give a gift and have tea with them, if it's a more peripheral friend/class invite, we go if it's convenient, but if it clashes with classes (which I have paid for, such as swimming or ballet) or another big event (family member's birthday) we don't go otherwise our entire lives would involve going to birthday parties of small children and could not include normal activities! I don't think it's unreasonable to decline a party invite for any reason, as long as you remember to let them know.