Have been lurking more than posting for a while but wanted to send my sympathies.
DS is 4.5 and DD 20m. DS woke every 45 mins for 5 months and then gradually improved til he slept through most nights from about 8m. Still wakes with nightmares but only once a month ish and after a horrific 6 week battle of return to bed age 2 when he went from cot to bed, he's settled beautifully at night, going off within 5 mins of us leaving the room. He dropped his nap age 2.5
DD woke every 2-3 hours and I congratulated myself on having a better sleeper.... He got better, she got worse! It's the messing on at bedtime that kills me too. Often an hour of up down, potty, drink, cuddle. Then wakes every 30 mins some nights and about once a month does a freaky sleep through which I can't understand. Drives me crazy. She won't have DH at all, screams as if being murdered. She now only naps for 30-45mins, which must be before 2 or she won't sleep til 9. Her molars are coming through and she's potty training so wakes for wees too twice a night (I know some would see that as lucky, but...).
Crying techniques are not my preference but I'd try it now if it wasn't for disturbing DS. Both have had a consistent bedtime routine from 6 weeks old.
So in case it helps, this is what we've tried:
No cry sleep solution - helped with ending bf but that was at 8m. Didn't help further than that.
Gradual retreat using this method - helped in that I can now sit by the bed instead of cuddling but can't get past that step.
Return to bed super nanny style - 6 weeks religiously by both me and her dad alternating. Lots of screaming and night wakings got worse.
Moving into a proper bed age 17m. Definitely improved things for a while. She hated the cot and the restriction of sleeping bags.
Couple of points you'll prob have considered. It's peak age for fear of the dark/nightmares so perhaps a night light? And separation anxiety is def our problem and is made worse by crying/leaving techniques, as we found to our cost, so I'm working on more one to one in the day, which is v hard when working and with another child, let alone with a new baby as you have!
Naps - DD has had a few days without one. Would you consider movie time as quiet time? Sit under a rug with a drink of milk and watch a Disney movie in a darker room? They'll either have a sit down rest, or fall asleep. At least you get a break (ish). I wouldn't force a nap, even if overtired at that age. They're too quick to know what you're doing and will fight every step IMHO.
Personally, I don't think I'd want to be too hard when there's a new baby, but you'll know how she's doing. Would you consider a bed? Combined with explaining about return to bed. Warn last drink/cuddle/book etc then rigid return to bed. Took 6 weeks with DS but my goodness it was worth it. If DD is still the same I'll be using the summer hols to do this with her, when DS can get a lie in. And I have no idea if it's he right path or not.
I've read so many sleep books and tried so many different things. I do think that actually they just get there eventually, but the bone crushing exhaustion is horrid, and I hate the way it makes me a cross shouty mum sometimes, because I'm too tired to think straight. Cannot imagine how you feel with a newborn too.
Probably not a lot of help but wanted to say you're not alone with an older dc who just won't go to sleep!
Sorry that was v long 