I think another crucial question here is who is your brother's Next of Kin?
If it is you, then I think you have every right to discuss this with the agency supplying your brother's care. If you find that their policies are not matched with your expectations, then you need to find another agency or ask SS to, depending on the provider of the funds (ie. you or SS).
If, however, it is someone else, then it is their decision and you should mind your own business.
I have SS support for my DD. That support comes in the form of agency carers, who come in for 2 hours at a time, twice per week.
During that time, I make the carer a cup of tea (or three), chat about her family and mine, what's happened during the week etc. She doesn't actually do a great deal, because what our family needs is an extra pair of hands with DD1.
That extra pair of hands actually diffuses DD1 and it means that DD1 doesn't get so wound up that she hits her sisters or causes a danger to herself/others. It's catch 22 - with the carer here, she doesn't need a massive amount of support because her needs are met. Without the carer, it's very obvoious that she can't cope with sharing my time between her and her younger sisters.
Now, I suppose I could say 'right...all you're doing is sitting around having a cup of tea and a chat.' Or, I could see what is really happening - the carer, who I treasure, is blending herself into our family, so that all 3 children see her as a normal extension of our family and are unaware that her presence isn't 'normal'. She is enabling DD1's siblings to grow up without the fear that they'll be hit by DD1 when she can't cope, because the carer can diffuse DD1 while I keep the other 2 occupied, or vice versa.
A lot of caring is being present and available. Your brother has as much right to peace and quiet as he does to active assistance. Imagine the alternative - your brother being constantly bothered by carers who want to show that they are being useful...it would drive me mad!