I'm a two-starters-and-pudding type 
I think it depends on the menu and the venue. If your starter could be an accompaniment (eg a tomato salad) then noticing that nobody else is having a starter and saying "oh please bring the starter with the mains" is considerate. But that doesn't work with soup or garlic mushrooms or something.
Often though I don't want a starter because there's nothing on the menu I fancy enough (allergies limit choice) so I'm not going to order something just to push round the plate to keep someone else company.
I don't think it is rude precisely but in a group of totally considerate people it would just never be an issue.