I've name changed for this for obvious reasons.
When you say challenging can you elaborate on the behaviours? Because my eldest (almost 8) has a dual diagnosis of autism and ADHD and his behaviour makes all of our lives a misery sometimes, and that includes our neighbours.
Shouting for hours, screaming, throwing things, smashing things up, attacking us, constant threats to do XY&Z. His demands are off the charts, if he doesn't get something he wants he will raise hell for hours until he burns himself out. Nobody sleeps. He has broken 5 televisions, 11 tablets, countless items of crockery and toys - both his and his siblings. My daughter is scared out of her wits when he comes home.
Contrary to what people would assume when hearing about the behaviour, we are good parents and our other two DC are a testament to that.
Sadly we are left with no other option but to medicate him after years of trying (and failing) to help him in every other way possible. He's in a special needs school with a permanent 1-1 and he's exactly the same there. Its an award winning SEN provision and they struggle with him too. He is quite possibly the most challenging child I have ever known.
All of this to say, I myself have said similar things to the DH in this situation IE giving up on DS completely, out of earshot of the DC but I say it all the same.
I do love him but I don't like him much of the time. How can you find a child endearing when they spend all of their time telling you they hate you and destroying your home? We are all walking on eggshells every day.
The DH likely needs therapy. I know I do.
I wish you the best of luck going forward, you sound like a good mother.