You can definitely get away with a more outlandish name as a middle name imo. I'm not massively keen on place names for humans myself. Astoria isn't a million miles away from Aurelia and I've definitely heard worse middle names...
Aurelia is nice though, have you considered Aurelia?
I think you should go for it OP, if you both love it. It does sound like a name iyswim. I probably wouldn't have it as a first name, but think you can get away with a lot more with a middle name. It's easy to spell and pronounce too.
@Rockaby thanks. Me too I've had a lot of names shot down for being too "posh" as well. I think with a quirky middle name you can hide it if you like. Our DD#1's middle name is Lana which I love but lots of (very unhelpful) people have pointed out what it spells backwards 🙈 Not so much of an issue with a middle name as a first name.