I think that your best bet is, rather than treat animals as a twice a week entertainment for a large group, to look at what the animals need first - regular care and attention. And, whilst you might not have planned it that way, give the animals the responsibility they need - ensuring that there is no way litter can be left or they can be fed inappropriate foods, see them daily, feed them, check their water and their enclosure (goats are bastards for a. Getting out of Fort Knox, b. Finding shit to eat that they shouldn't and eating it and c. Dealing with issues/perceived threats by meeting them head on - literally), cleaning them out.
It might be better for the goats if you had smaller groups each day, what with animals needing care daily. That way, you could pick children who would work well together and, rather than have 'the autistics' on Mondays and 'the normals' on Wednesday (and that is how a lot of the kids would interpret your lumping ones with particular SEN together), you actually look at their personalities and strengths. And if there are any children who don't see it as equivalent of going to Chess Club once a week and want to be there daily, whatever the box on SIMS says, welcome them with open arms.
To give an example, at my last place, there was a 'Social Skills' club that every kid with SEN had to go to. Which really didn't go down well with a couple who not only were extremely articulate, they didn't particularly enjoy being forced to stay with the same kids who were pissing them off in class and in the playground - particularly ones with ADHD, who they found incredibly difficult.
Put five kids with ADHD into a group and they could be bouncing around to rival the goats, rather than being calm and talking to each other about how hard it is to do their maths homework whilst simultaneously pissing off the ones who manage quite well with medication - and multiple diagnoses are common; what group do you shove the Autistic kid who also has ADHD? Some might have a huge problem with mess and smells, some might not give a shit about that where an animal is concerned. And then there are the kids who have physical/mobility difficulties - do you shove them in with kids who don't do taking turns or being calm and quiet, or do you decide they don't count as SEN, so don't get to come along unless the 'normal' ones are there to take over?
What about the more vulnerable kids who don't have a diagnosis?
It's not unreasonable to have the animals there (although the number of attacks on school animals worries the fuck out of me), but I think YABU to look on them purely in terms of being a means of putting children together to talk about stuff that's entirely unrelated to the animals.
The animals and their needs MUST come first.