I don't think you're in the wrong at all.
I might have done exactly the same depending on the situation.
DD1 is very calm in situations like that and, on the occasions that she's been to A&E recently, DH has taken her. Unless there was something distressing about the treatment she needed, I wouldn't go rushing over.
DD2 is a different kettle of fish and DH struggles to handle her at the best of times. If it were her, I would go straight away.
Any problem that involved a lot of blood would have me there PDQ too because DH panics at the sight of it.
I know my children and you know yours. You felt that your DS would be fine to have his arm put in a cast without you but in his dad's care. That seems perfectly appropriate to me.
What you didn't factor in was that your DH expected you to be there, probably because he was finding it stressful and he needed your support. He didn't bother to tell you this. He just assumed that you would realise and arrive.
This isn't about your maternal instincts or caring for your DCs. It isn't about whether your DH can or can't cope with all the aspects of parenting you can.
It's about a breakdown in communication and your DH being stressed. He didn't tell you that he needed you and he didn't tell you that your DS needed you or was distressed. My 14 month old pushed a toddle truck round A&E for four hours with a broken arm and a big grin on her face. If he had been asking for you, I'm sure you would have left immediately.
I hope he's apologised for his horrible comment and that you will treat it as a moment of madness and move on.
I think those who are criticising you because you didn't drop the phone and race to your child's bedside perhaps have children with very different characters or haven't experienced more serious illnesses and injuries in their children. An uncomplicated broken arm that is put in a cast then the child is immediately discharged pretty minor after all.