I was involved in a committee, which had benefits for me in terms of being allowed certain advantages (first in line etc), however, the benefits did not outweigh the negatives.
One guy I worked with on this committee did absolutely fuck-all, messed up things I'd organised, made lame jokes and started goading me on Facebook over our opposing political views. He started posting very right-wing and factually incorrect memes and when I (and others) called him out on them, he started blocking us from seeing his posts.
I realised I was feeling anxious about being part of this committee, and the pressure it put me under, all the time. So I decided to resign. I couldn't tell the Chairperson the real reasons about Toryboy, as I think he would have been binned off in favour of me, so I just stated that my work was becoming too stressful and I 'couldn't give them the attention they deserved'. Worked a treat.
My point is, once I'd resigned, the worry about resigning obviously instantly disappeared, but more importantly my anxiety about being on the committee immediately lifted. It was the most enormous relief.
I'm still good friends with the Chairperson, and have blocked Toryboy on FB. And I feel so so much happier.
Resigning is often scary, but it's usually worth it in the end. Especially if you're being bullied and undermined. Good luck OP x