My dh has both Rays and Raes in his family. Rae is a female form (at least according to Behind the Name) as are Ramona and Raylene for Raymond. I like Ramona :)
For the OP I don't think it's just Isla Ray that's not good, I think Isla Hall isn't ideal either. Isla's a bit tricky, too word sounding I think.
Poppy is also surely very popular though? Or perhaps it's passed it's peak?
Elsie is pretty, and nice to acknowledge both sides of the family I think. Elsie Ray Hall could work, or Elisabeth of course - if you like Olivia because it's 'proper' then I'd not choose a cutsie sort of name for your second dd.