OP, it might help you get over your guilty feelings if you frame your refusal to take on your brother as due to a need to protect your family - your children and husband don't want him to live with you, it would cause enormous stress.
Even if your kids were grown and had moved out and were having families of their own, they wouldn't want to visit you if you had moved your brother in.
Your mother has made a rod for her back, don't let her pass it on to you.
He's not a good person to have around you or your children, so you need to stop agonising about what you would do if your mum wasn't able to 'look after' him.
A PP suggested that he'll most likely end up living in your mum's house for all of his life, as you and your brother will let him, to avoid having him live with you.
You need to have a serious chat with your mother to let her know that you and your husband don't want her and your brother being such close neighbours, you love her but don't want to live in each other's pockets, your brother winds the kids up etc.
If she gets thick with you, so be it, it'll blow over as she'll want to see her grandkids, better than you ending up moving away if she did buy the house, that would probably lead to a complete break.
You need to stand up for your kids, their failure to launch uncle is not a positive person in their lives.