I'm assuming this is a pretty small wedding. Only invited relatives from the grooms side seem to be mother, SDad, sis and BIL. Even if you add on another 10 friends of the groom, that's quite a small group and if the brides side is roughly equal, that's a guest list of 40 or so.
Which sounds my ideal wedding.
However, it seems a bit overkill to have 3 flower girls for a wedding of that size, let alone another 2 on top.
Maybe it's numbers that are restricting the guest list, not specifically targeted step relatives or nieces.
But whichever way you look at it, it's incredibly thoughtless not to invite the step siblings and the nieces.
Your brother needs to either acknowledge he's agreed to the budget and the consequential number restrictions. Or he needs to stand up for his relatives and argue the case for them to be included in his special day.
OP and her mother have bizarrely overstepped by assuming the girls will be flower girls. But brother created this situation.