OP, if you like them that’s all that matters. I think people are being unnecessarily rude here. I don’t own loafers because they don’t suit me. But if they do suit you, then go for it. Maybe you could wear them with a navy fitted dress, or even a shorter skirt and top. I’m sure you can make them work. Don’t dress for other people, choose your own clothes and shoes that make you feel good.
And clothes are like baby names. You should not ask people’s opinions before they arrive. People are usually less harsh once the clothes or baby have arrived!