I'm an au pair at the moment, with a really great family and it's fantastic, but I have also had a bad experience with au pairing.
I think part of the confusion comes from the fact that au pairs are supposed to be treated as and act as part of the family, but with other duties, and, after all, there is still payment involved.
A lot of it depends on the individual family and au pair, because although there are guidelines on appropriate hours/duties, when an au pair becomes part of that family they might be expected to do extra things as well - which, to a particular family would be perfectly normal, but when it's not really outlined in the contract it can be a bit difficult.
I can only speak from my own experience, and that of some of my other au pair friends, but the cultural difference plays a big part, too. For example, I found that the country I au paired in during the summer (and many other people have said the same) tends to have the view that au pairs are there just to earn money, and not to learn a language/experience a new culture. So, when the aims of the au pair and the family are different, that is difficult.
I suppose it all boils down to the fact that it is still quite a casual arrangement - yes, there is a contract (not sure if it's legally binding though?), but most au pairs are teenagers/early 20s and don't have much life experience.
I just thought I'd add - I've seen a few posts of people complaining about their miserable au pairs - it is so difficult to adapt to a new country, culture, language and family, and also to be assertive about what you need as an au pair - sometimes they do just need to be a bit miserable! :)