If someone has reported him as violent or police have been at all involved, ss will have done a check on his kids.
DP got drunk one night and was really nasty, throwing things etc. My neighbour rung police as we were arguing in the street (long story), ds wasn't here, he was at mil's. I got a letter to ring sw, agreed a day and she called for about half an hour, spent most of it talking to dp about alcohol not being a solution, managing stress, etc. Also had health visitor out a few months later (both were really nice) and she made it clear I had to be able to put ds first, and be aware of any potential threats or dangers , but seemed happy with things, ds healthy and happy and doing well at nursery, alcohol banned from the house etc. Got a letter a week later saying I wouldn't hear from them again.
You might have similar if he's registered at your address. Difference is, you are there to supervise, his ex is not. The sw did suggest no unsupervised contact for a few weeks, (I asked if mil, his grandad, his aunt etc could supervise, and she said that was fine, as long as I trusted them).
It sounds as if dp and the ex don't get along/she doesn't trust him. Has he been to court for access? If the court denied it, do you know why? Was the relationship volatile with lots of shouting? Maybe she has seen him in a darker light than you have yet.
Apparently relationship counselling is not advised when one of the partners/spouses is in anger management. Dp has had counselling, and covered his anger as part of it, (and where the real anger comes from, and why trivial things can be the straw that breaks the camels back) and it seems to have helped. He's been diagnosed as having severe anxiety, which makes so much sense in hindsight. Maybe he knows he needs to sort himself out before he tackles seeing his kids again. 3 years is a long time when you're a kid.