It hardly seems right that your mother (according to your brother) wants you to give half of the entire amount to him, who hasn't contributed a penny, which would mean he would get all of her contributions and some of yours which rather defeats the object of having this joint account in the first place.
How and why does your brother know about this account and has he suggested to your mum that this is what should happen? Why does he want this money now?
Check with the bank if all monies would come to you as the surviving account holder when your mother dies and if not how it would be split. It might make a difference if both signatures are required for a withdrawal or just one. If all the money would automatically come to you this could be why he is pushing for it before your mum dies.
You say she can't remember where the money came from, so if she has dementia or similar this would make a difference as to whether she has the mental capacity to make this sort of financial decision. Does anyone have power of attorney?
Personally I don't think your brother should try to assume any rights to this money as it was an arrangement between you and your mother, but what is morally right and what is legally right are often very different things.
It's not really about the amount of money, at best he should only expect 1/2 of your mum's 1/2, so 25% of the total amount, but it is the principle. WHY does he think he is entitled to it?'
My mum's will was set up so that it was split 50/50 between my sister and me. My mum died during covid (not from covid) and she had wanted to change things so that I had a greater share as I was the one who was her carer for 15 years and she wanted me to be secure. I had to assure her that whatever happened I would be fine. My parents paid for my sister's wedding and whenever she wanted/needed anything my dad stumped up. She had much more from them but still expected "her" 1/2 when mum died. Because the will wasn't formally changed my (now ex) sister insisted she got her pound of flesh half even though she had a house, mortgage paid off and I had lived in my mum's house as her carer, spent all my savings on things for the house and was about to become homeless. She even had the temerity to say that I wasn't entitled to what I did get because I had lived with my mum "rent free". I didn't see her changing continence pads and generally dealing with stuff at 3 in the morning.
As the saying goes Where there's a will there's a way, but usually it is Where there's a will there is some sort of family feud or argument or any amount of ill feeling.
The fact that she suggested you had a joint account, in the same way as her mother had with her, says that if was for you and your future. If she had wanted your brother to be in on it he would have been named on the account or she would have had a different one with him.
Ask the bank in the morning but don't just hand over the cash on your brother's say-so.