YANBU. It sounds like your DH relationship with DS and with you really needs some work.
Saying really inappropriate things to DS, making excuses not to play with him suggests that a really good non-blamey talk between you is needed to establish how he is really feeling. He clearly doesn't have very good coping mechanisms. Perhaps he is feeling inadequate and insecure.
We often expect the adult in the situation to be able to cope and have a willingness to learn to cope (quite rightly so) BUT......from what i know about attachment and adoption (which isn't a massive amount tbh) there are lots of complex problems that can arise.
You sound like a fantastic Mum but maybe your DH needs some support to begin to get things right with DS.
Men are not encouraged to talk about there feelings, have no-one to turn to when they themselves are struggling with relationships and parenting and psychologically withdraw, adopt unproductive coping mechanisms and can feel very isolated.
I'm not saying that your DH hasn't been a prat but it might just be that with support and someone to talk to you can get on track and start feeling happier.
Children with ADHD and additional needs are very challenging and without support things can be very difficult. Men are terrible at seeking support and this can lead to family breakdowns.
Do gently begin to explore some of the issues with him if you can. Perhaps see how he is feeling first.
Perhaps he feels like a failure? Men often take longer to come to terms with dx of an additional need than mothers do. They have complex feelings about their roles, parenting and masculinity and rarely anyone to really talk to.