I am really very surprised to see how few people have an issue with this. We were taught that basic tablemanners dictate that you don't start until everyone has their food. Just like we were taught to chew with our mouths closed, how to use a knife and fork, not to stretch over people at the table etc. That doesn't mean we don't break the rules sometimes, but everytime I eat out with friends or for work or at someone's house, I am grateful to my parents for teaching me the basics.
However, I would also argue that as the person serving up, you should make an effort to get it done and not faff around. It drives me mad when DH serves and then keeps having to wander off to get things he's forgotten. Similarly, as children, if we were responsible for setting the table, it was considered pretty poor if we forgot obvious things like napkins or salt/pepper which would delay the meal while someone scurried around picking up whatever is missing.
The exception, is that sometimes, at dinner or a restaurant, the person serving or whose food has not turned up, may say to the rest of the table, "please start, don't wait" and you shoudl do so in that case I think because otherwise it just makes the person delaying everyone else feel very very self conscious. DH disagrees and will continue to wait, so that sometimes gets tricky with us! 