Hold on in there, it's not always like that (although more common than people will admit). I was in the same position for eight years. Yep, I'm a mug. I was exactly the same as you, loved it whilst I was there, but hated everything else. Our leader in charge finally retired and it all changed overnight.
I completely agree with badguider - assuming your DC is understanding of the situation. One of the reasons that I had to stick it for so long is that the various DCs we had over that time all sympathised, but would do nothing to actually help me. It sounds like you're in a better position as you've had mediation, albeit unsuccessful.
I hope the next bit comes across properly when written down, I'm not trying to excuse her behaviour at all.
A couple of years after she retired, I took over as Unit Guider. There is so much to do! Although I had done Module 4 of the ALQ, it wasn't any real preparation for doing it on my own. I had two adults that volunteered to come and help me, and I was so grateful, I tried to do everything myself in case it put them off staying. They are both now here to stay, but I'm finding it hard to let go as so many of the processes and jobs seem to be interlinked.
I can see myself in some of the behaviours you describe (although not to the same extent!). I get so stressed that my unit will be badly thought of in the District that I try to overcompensate, which is another reason not to let go. I hope I hide my worries well enough that the other leaders don't know, and as I have to do a lot of negotiation at work, I hope I treat people fairly.
Don't let any of that put you off being Unit Guider for the other unit, I couldn't give up Rainbows ever, I've voiced the inner fears (bearing in mind I am on medication for anxiety as well) rather than day to day genuine problems.
You've got exactly the right perspective, it's supposed to be about the girls knowing and growing (or whatever the new strap line is), but that doesn't mean it should make us miserable. Remind me of that after I've been to a noisy meeting and come home begging for wine!