Sounds like your brother (and possibly your Dad since that is where he gets his information from) don't really understand your son's condition(s).
Fortunately, you DO understand him, his abilities and his limitations. And you are providing that for him. If you can, leave it there and ignore them.
But if you can't leave it there, you need is something to tell your Dad - either for him to say to brother when he mentions it or something to filter out what Dad hears from him.
Something along the lines of "Dad, if Brother talks about my son, remind him hasn't even seen my son in XX years, how would he know what he can and can't do in the workplace? I am not interested in hearing what Brother says about my son, please don't discuss him. DS would find it very distressing to find the family talks about him behind his back. "
Or "Dad, Brother has no qualifications in this area, he's not seen DS in X long, so really he has no expertise here. I am following the advise of his medical specialists and my own understanding of him, please don't talk about him to brother, and don't tell me what Brother says, as I find it very upsetting to hear his unqualified opinions."
Or "FFS Dad, son is going to face all kinds of prejudice from strangers, don't you think his family can be supportive of him (and me)? I really don't care what Brother thinks, he's not even seen X since Y. I am following my medical specialists advice, as well as my own understanding of him. Just don't pass on Brother's opinions to me, I am not interested in what he thinks about me/DS."