When DS1 was c.12m I started going to a toddler group where a small group of us, who had all initially come on our own, became good friends.
One of the Mums had 2 boys who were slightly older than the other children, large for their age and "boisterous". As all the kids got older these two boys became a problem particularly for the girls, but this was the mum I felt I got along best with.
Anyway some of the others started making plans which excluded this family, which I felt uncomfortable with, so I went out of my way to make sure we did things with them. It turned out she had suffered from PN depression and was still suffering on and off. She was aware that this affected the quality of her parenting, but didn't know how to improve matters. She needed support from her friends.
Now 6 years on we are still great friends. She feels much better and although she still has off days, she says she feels normal again. Her boys are 8 & 10 now and the eldest is fantastic with my 4yo. She is the one person I could always call on if I needed someone to help me out with childcare at short notice. We have lost touch with the rest of the group.
So I would say, stick around if you can. If the children really can't take it, then have a break for a while and maybe just meet with the other mum, perhaps for a drink in the evening. As the children get older, the dynamic in the group will change and you may find it's yours that become the problem! This happened to me, as DS1 can be very strong willed and hers didn't want to play with him for a while, but she never judged and was always supportive.
I've waffled, but I hope you get the gist. I'm very glad I didn't give up because her boys were a problem. After all you're entitled to your friends. They don;t all have to be for the benefit of your DC's.