Re the 'unpronouncable' - I agree. The sound (final e, acute accent) exists in several Scottish accents and of course is originally French. (The first-known person with the name was, as previous poster said, a man and French-Scottish and a close favourite of Scottish James VI/James I of England, who wrote a poem about him comparing him to a bird of great beauty.)
But the final 'e with an acute accent' is not a natural sound in many English accents. It's halfway between (sort of) English Esmay and Esmeh.
Also, as a previous poser has said, Esme (with accent) is the MALE version of the name. To be correct in French, it should end 'ee', with an acute accent on the second e (out of the three).
Personally, I prefer the alternative spelling, Edmee, for a girl. In that, there is also an acute accent on the second e. It was the name (Edmee Elizabeth) of the witty and socially-critical feminist author EM Delafield.