There are so many names in the world that it would be sad to settle in one you don't like. However, you need to have an idea of what name you do want (along with your dp).
Agree with this. There is nothing wrong with changing his name at an early stage though, if you really dislike it - it isn't that big of a deal (Is he even formally registered yet?). Leaving his "old name" as an extra middle name is a nice idea too.
But if your DP loves the name he's chosen, can you say what you don't like about it (ie. is it too old fashioned, too popular, don't like the sound etc?) Or can you tell us your daughters name, then it would be easier to make suggestions that might help (as we would know what your style is/ what would "fit") ; as Edward, Rupert and Albert are fairly different styles imo - although they have a theme of being traditional and are two syllable (I would count Edward as a true classic, Rupert as more quirky, and Albert as a retro/vintage name).
True classics like Edward (top 100 for the last century) are James, Alexander, William, Thomas, George and Charles.
Other well known or classic 2 syllable names you might like (some are more quirky or retro and a couple should be 3 syllable but are often pronounced as 2): Samuel, Henry/Harry, Michael, Jacob, Alfred, Frederick, Adam, Rowan, Isaac, Rufus, Joseph, Lucas, Marcus, Robert, Matthew, Owen, Oscar, Felix, Evan, Maxim, Leo.