All the Millies I know were Millennium babies. I think people don't like the name much because it seems very much a name of its time. It seemed to come from nowhere in 2000 and has been popular ever since. You'll find people dislike faddy or trendy names on MN.
Millie sounds very pretty, as I'm sure most people would acknowledge, but it's so commonplace among the current generation that it puts a lot of people off. It's this generation's Claire or Jennifer or Sharon or Tracey.
Another qualm people have with names like Millie is that they were part of the 'cutesy' trend - Evie, Ellie, Gracie, Maisie, Daisy, Alfie, Harry, Freddie, etc - that was popular in recent years. There's been a recent trend, I think, towards more substantial/formal names - Evelyn, Eleanor, Grace, Margaret, Alfred, Henry, Frederick - and perhaps using the diminutives as nicknames.
'I'm thinking of naming my daughter Camilla/Millicent/Amelia and nicknaming her Millie. What do you think?' If you asked that question instead, the response would be overwhelmingly in your favour This is a quirk of MN and not necessarily representative of the 'real world' where Millie by itself probably outstrips Camilla and Millicent massively in popularity. That said, it might be indicative of future trends and you'd be wise to be aware of them even if you choose to follow them.
My advice is just use it if you like it but beware current trends.