I started a thread a while ago asking about all sorts of names and this wasn't on the list. (I had Barney, Otis, Zachary and Finley and also Leonard for anyone who is interested!)
We have arrived at this as a compromise name but it's actually really growing on me. What do you think of Sam to go with Rory? He would be Sammy to me as a baby (although dh wouldnt use Sammy I don't think) and then Sam or Samuel.
I think it's simple and classic and I've never met a Sam who isn't nice.