It doesn't sound like membership is too much of an issue, if being down to 4 is as bad as it has got.
I run one writing group, and I'm a member of a longer-established one - I started one while on the waiting list for the other.
It took us several months to get off the ground, with three of us hanging in through the barren early months, and now we have a waiting list. We put a card up our local independent bookshop, and started a meet up group, a Facebook page and sent messages to local writing tutors and other groups, asking them to send people our way. We got 4 members from waiting lists for other groups, one from the bookshop, one from a local writing event, a couple from the meet up group, and a couple more from word of mouth.
Our meetings have ranged between 3 and 10 people, and actually it's the smaller meetings that are sometimes the most productive. I think about 5 or 6 is about the best number for us.
It sounds more as though your meetings need a better format, and some ground rules. We circulate 2 or 3 pieces of writing (max of 3000 words, but we'll occasionally look at longer pieces) a few days before the monthly meeting, giving people a chance to read the pieces and make some notes. We then discuss them at the meeting. The person who is being critiqued, agrees not to speak until the end, unless there is a specific question that needs answering.
The whole purpose of the meeting is for feedback, and we always have plenty to discuss, because we've had a chance to read and digest the pieces in advance. People tend to bounce off one another, and everyone tries to be as constructive as possible.
We broadly take it in turns to submit work, but sometimes people won't have anything, so someone might finish up submitting 2 meetings in a row, or not for a while. If we're short of work, someone will generally find something for us to look at.
The beginning and end of the meeting is just general chit-chat.
Might be worth trying to change the structure of your meetings.