Ideas for solutions to help much appreciated. (Elderly DM and neighbour with dementia).

I would suggest you have a conversation with the children, ideally not with their dm present. Chances are she is underplaying what is happening, that he is wandering and intimidating your dm and how often. Sounds like he really needs close supervision when she wants to go out. Is it happening while she is inside too? Maybe the caring is too much for her too? If no progress it needs to be social care involvement for his sake as well as dms.

We have a neighbour with dementia. He often comes and knocks on our door, they haven’t lived here all that long, 5yrs maybe so didn’t know them when he was okay. He babbles like a toddler, very insistent with whatever hes saying but you can’t understand him. He gets quite agitated at times. I used to answer the door, but now iIf i see him on the cameras I just ignore it and he wanders off. I’ll text his wife that he's out wandering.
Could you install Ring type cameras so your mum can see if he's there, and if hes knocking at the door she knows to ignore it.
How is he getting in the garden? Would a better higher fence or letting the hedges grow higher help? Locking the garden gate? Do your best to fence him out basically! I know you shouldn’t have to, but worth it to make your mum feel safe.

Escalate with adult social services now. Let them make the call.

Raise a safeguarding concern with Adult Social Services. You may not get very far with them, but it's worth raising.

And help your mother to do whatever she can to protect herself, a camera is a good idea but if he can be physically prevented from getting to the house, even better (by padlocking the gate, or putting up a fence). Obviously she can't lock the gate all the time as others need to come and go but it would be worth having the option.

People with dementia get fixated on things and he isn't going to be talked out of it, and clearly the family are not going to keep him indoors. So she needs to do what she can to protect herself.

Obviously if this escalates and he does try to touch her or force his way into the house, she must contact the police again.

Yes, I would do this too. Obviously they are trying to protect their DM but that can’t be at the expense of yours. I’m not a shouter, I’d go in calm but firm, but I wouldn’t let them get away with it.

I’d seriously think about notifying Adult Social Care, you might be doing them a favour.

It probably wouldn't be helpful but I would go round when the children are there and give them all a rollicking for not looking after their DF properly and shouting at an elderly lady for being frightened. How DARE they! I would be raging tbh OP. Give them a taste of their own medicine.

OP, your DM should not have to be a prisoner in her own home in order to avoid being pestered by this neighbour. Yes, it’s very sad that he has dementia, but that doesn’t give him the right to be a nuisance to others in the road. Is he actually safe to be left home alone? It doesn’t sound so. I would be approaching SS or the Council and raising the issue of his welfare, as well as the upset to your DM. It’s not at all OK for an 85 year old woman to be terrorised by a man with mental health issues, no matter how difficult that may be for his family. Someone needs to take responsibility - perhaps you need to suggest to neighbour’s grown up children that rather than upsetting your DM who has done absolutely nothing wrong, it’s time they started to think about some sort of care or nursing home for their DF.

I think I’d raise this as a safeguarding concern with the local authority