I wonder whether it's worth asking on the Adoption board, as people there often have experience of helping young children process information about birth parents who made challenging choices. One option used is to create an "All About Me" lifestory book, in an age-appropriate way, about their current and past life (if you google Joy Rees there is some good advice and examples of how to create these books). In those books, and in training given to foster carers and adopters about talking about tough experiences to children, sometimes we're advised to talk about what a child needs (food, drink, cuddles, play, comfort, safety...) and that all children need all those things. That it was very sad that mum wasn't able to give you all those things, and you weren't safe. Sometimes we don't know why people find that hard, but they do. Sometimes people are a kind of poorly, which makes it hard for them. But children need to be safe, and that's why you live here with us, and you can talk about mum whenever you need to... You don't need to have all the answers, but just be age-appropriate honest. It's OK to say you don't know, but that you know it's really hard for him. You can validate his feelings about it "It's OK to find it really hard to understand, and to sometimes feel sad, or angry or upset about it." Caro Archer's "Parenting the child who hurts" is also good for games and activities to help a hurting child grow in a new relationship.
Children who have experienced neglect/trauma are often much younger emotionally than their biological age, so can need lots more help managing their feelings. In the situation you describe, it's also common for the child to have a deep need to control, because they have experienced the fear of being in a situation out of their control, with violence and helplessness. That can be why "no" is so very hard for them to cope with, and sometimes needs to be presented differently. Clearly, you will still enforce your chosen boundaries, but sometimes using controlled choices helps - instead of "It's time to stop playing and get dressed", you can try "Do you want to put on your trousers or your t-shirt first?" The theory is that they get to use their control, but you still get the task done. Also used in "Would you like to walk or scoot to the shop?" rather than "We need to go to the shop now."
You say he is "incredibly affectionate and not very clingy" - indiscriminately affectionate to people he doesn't know very well, would you say? Because that potentially sounds like disordered attachment to me, which would be totally understandable based on his early life. A learned behaviour to form quick bonds with lots of people, because he cannot rely on the ones who should be keeping him safe. Clearly that might be way off as I'm a stranger on the internet, but it is a waving flag, and I really think your HV or GP should be listening to you more.