It could be extremely relevant - I worked at a very poor school (it closed in the end) and the bullying was both horrific and very specifically targeted at the children's vulnerabilities. Examples directed at different children included;
Oh, it's Mother's Day soon. Fred can't do anything for his Mum since he killed her being born.
What you getting your Mum for Christmas? I know, a big bag of smack.
They'll send you back to a home the same way your real parents didn't want you.
Was your real Mum a whore? Was that why they took you away from her? Or was it 'cos your Dad was a paedo?
Nobody loves you, your Mum didn't want you, your 'Dad' doesn't want you, she'll get rid of you next.
Bet your Mum wished she'd had an abortion when she saw you. That's why she gave you away.
Are children's homes like Battersea? Living in a cage where people come and look at you to choose one? (from another kid 'they should have picked a puppy instead')
You got abandoned because you're weird/a freak/creepy.
No wonder your Mum didn't want you.
Did they keep the kids they wanted and got rid of you, like the pick of the litter?
(school lesson on Frankenstein - 'Bet that's how Fred's Mum felt when she saw him')
And then there's the vulnerabilities for manipulation and lies, rather than outright abuse - 'it's hard when you're all alone in the world, we can be your family, not those people who bought you like a toy'.
He could also see breakdown in adoptions with other children and fear it happening to him - it's very common.
We jumped on any adoption/LAC/bereavement related abuse like a ton of bricks. But barring reintroduction of a Scold's Bridle for the kids who do this - and you have to bear in mind that some also come from awful circumstances so they're using their own fears and adverse experiences as a weapon against others - the damage is already done by the time the words come out.
The other potential issue could be what is his father saying to him? Could he be saying really hurtful things, blaming him for the divorce, suggesting that if he doesn't 'sort himself out', you won't be there for him, threatening to put him back into care, saying that DS owes you both good behaviour because you adopted him and he should be grateful for it? It's horrible to think about, but it is a possibility (I've heard awful things from adopted parents, too).
Being an adolescent is hard. Being an adolescent with SEND is hard. Being an adolescent with SEND and a knowledge/fear that fundamentally, you aren't loveable because the person who is supposed to love you rejected you, didn't care for or protect you and either decided to give you up or had you taken away from them is really, really hard because it bites to the core of your existence.
That's why adoption is very relevant and not rejecting him to acknowledge it.