I second the recommendations for Kodaly type programs; it's brilliant for ear training and aural skills and helps lots with learning an instrument later. Also Suzuki is good for the little ones, though I personally would prefer standard lessons from about age 6 or so, less emphasis on rote and more on reading music, but many people love the Suzuki system and it has lots of benefits as well. I would definitely go for private lessons by 6 or so, as there is so much about posture and positioning that can make a real difference, and bad habits can get ingrained.
Children who start a bit older might well catch up in terms of ability, but it doesn't mean that time or money is wasted on the younger children if they enjoy it at the time! Also getting confidence, muscle memory, enjoyment of performing etc can help.
And the children who are slightly more advanced at younger ages then seem to get more opportunities to do things - chosen for orchestra or in higher level groups or to play solos or whatever - and that they feeds on itself and they become better, more enthusiastic etc, not because they're necessarily more talented, but just being better at the younger ages can sometimes bring opportunities that are less available to children at the same level who are older.
A really good choir is fantastic for later musical training, if you can find a good children's choir or church choir. You don't have to be at all religious. It helps them learn to listen well for tuning and hold a part and understand basic harmony, rhythm, following a conductor etc. And can expose them to good music early. Playing lots of music in the house all the time is also helpful, so it just becomes familiar - classical, folk, nursery rhymes, carols, anything.
There are lots of concerts by orchestras for children that introduce lots of different instruments, and children can get a feeling for the sound that they like, to help them choose which they want to learn later.
Piano is great if they seem to have good co-ordination, as it does help learning harmony and understanding theory later too, but some children find that learning single-line instruments is easier to start with if they are struggling. I'd personally go for cello as the second instrument, but it depends so much on the sound the child likes! (I like the lower sound for strings, not so screechy, and cellos still get good tunes as well as nice bass lines in orcehstras, or it can lead to bass, which can be in all sorts of ensembles and styles). Don't write off harp though, as you can always get a small harp and go right up to grade 8+ on it these days, nowhere near as expensive or big as the ones you might be thinking of!!