Im very much its their day, they dont have to invite anyone etc etc, but I think in what youve posted that your brother is a selfish knob.
So his nieces arent invited, but his wifes cousins children are going to be flower girls?
Im not surprised your DH is angry, I would be fuming too (& I did have a child free wedding, but it was child free for everyone and the 2 that did have kids lived where we got married and grandparents had them for the day - no overnight stay required). Its either yes to kids or no, not yes to a distant relation of the wifes but no to a direct blood relative of the grooms.
I also think its incredibly rude to not invited his step siblings - especially as he went to their weddings and can understand why your DMs partner isnt going.
I get the feeling that once the wedding is over, you wont be seeing much of him anymore.
And tell your mother NOT to pay for the flower girls dress, that is taking the piss. I think you need to speak to your brother and spell out exactly what a shit show his wedding is and how he has completely disregarded his own family and that hurts. His reaction to hearing that will tell you everything you need to know.