It sounds like all the parents in your group, even those who do occasionally help out, need to hear exactly how you're feeling.
In your situation, before jacking it all in, I'd hand out something like this (I've mainly used your words from your OP):
Dear Parents,
I have now been running XXX Beaver group for over 5 years. I work very hard to make sure we have vibrant, engaging and exciting activities every week that link to all the badges and that all children can achieve in.
I spend around 5 hours a week on the admin, setting up, running etc of this colony, all for the sake of the 20 Beavers who come each week and really enjoy it.
I am an unpaid volunteer, a volunteer who has a full time job, children of my own, many other commitments. I do this because I really believe it benefits the children and I adore the children.
Over the years I have noticed volunteer support from parents dropping each year, and it is increasingly difficult to run the group without enough support. My own children have been and gone from Beavers, so I only do this now because I enjoy it.
However, recently I have been getting a stream of complaints about the nature and style of activities not being badge focused enough (all activities are built around badges), complaints if I organise trips, complaints that there is not enough adult attention for each child, and the list goes on.
The lack of support and complaints are wearing me down and sapping all the joy from this role, to the extent that I am considering closing the group.
If anybody has ideas for improving the sessions, then please volunteer your time and support me in the running of this group. Any complaints need to be backed up by an offer of help. There are currently XX children on the waiting list so if you or your child are not enjoying Beavers, please allow another child a chance to attend instead.
(You could maybe add another paragraph thanking those that do volunteer, if you feel it's needed)