DH thinks I should be able to just ignore it.
Why? If your son thinks the behaviour you've described here is acceptable, he's at risk. He obviously feels "safe" treating you like shite, but if he does the same to others he is running the risk or a physically violent reaction or serious negative consequences at school, in his social group(s), and eventually professionally. He's ELEVEN. Parents (both) need to guide him, actively, AWAY from harmful behaviour.
Further, his dad should NOT be speaking up in FRONT of the child criticising your reaction/parenting. Do you also do that to him? The two of you obviously need to discuss the matter, but in private.
It doesn't seem like your husband is able to listen to and understand your concerns, feel empathy, put himself in another person's place, or compromise at ALL. He just wants what he wants and shouts louder when he doesn't get it, all the while not being able to articulate his logic. That's a rotten basis for a relationship OR for co-parenting. It's ironic that he's telling you to act like an adult when he himself, from your description, seems to be stuck in adolescence. Is he a good parent or a good partner in ANY way? If not, it may be time to just sideline/ignore him on this matter and do what you can to help your son on your own. It sounds like your husband's upbringing, like yours, was detrimental - albeit in a completely different way - but that can't be your top concern now. Husband's an adult who can get help when he's ready, son is a child who needs you to help him get help.
From what you say here, your son MAINLY does this to you, occasionally to other household members (dad and brother) and rarely or never to anyone else. This means he's targeting you, specifically, and I think it's worth finding out why that is. Can the two of you go to family counseling on your own, maybe with the brother if appropriate? Invite your husband, but don't react if he doesn't want to go, just carry on getting your son help. A set of diagnostic tests to make sure there's no underlying physical, neurological, or MH issue before proceeding with it as as social/relationship issue would be a good first step.