Swedes - like you, I used to hanker after living on Place des Vosges (or on the Ile Saint-Louis) but I am completely over that now, probably because I lived very near the Place des Vosges for many years and in that time saw the whole area transform from a very peaceful and beautiful, albeit run down, architectural gem to a tourist shopping centre . The rue des Francs Bourgeois has been ruined - it is all Kiehl's and L'Occitane and Birkenstocks shops these days.
And the Ile Saint-Louis is being bought out by oil money.
I agree that the area around the Eiffel Tower is not very attractive. But then, I dislike intensely Haussmannian architecture - I love 16th/17th century architecture with a thorough modern makeover (there is no choice but do this to make it livable) and otherwise I like Art Nouveau or Art Déco^.
So - to answer your question - where would I move to. I have to take schooling into consideration, so need to stay in the Western arrondissements (where the bilingual and international schools are); I want, preferably, an interwar apartment (even better with a garage under the block - these were just beginning in the late 1920s); I don't want an area that is attractive to non-residents (because then the local amenities go haywire).
So - 6th; 7th bordering the 6th and nowhere near the 15th; the 8th bordering the 17th.