You are not a bridesmaid.
You could decline to wear navy - maybe say you'd already chosen your outfit. But, it's navy, not pink taffeta. Don't you have other uses for a smart navy dress?
Traditionally the groom's family hosts the rehearsal dinner (a counter to the bride's family traditionally paying for the wedding). Quite possibly not the case and all funded and hosted by the couple themselves. You could discreetly ask your brother about this.
But, the dinner is likely to be a late evening, grown-up, formal dinner. Why would you want to take tiny children to that?
They're trying to include your family by including your children. (The children may or may not enjoy it on the day but I bet they'll look back on the photos fondly later). You've been asked to wear navy (neutral, useful, re-wearable colour). All sounds fine to me. Say yes, or say no.
Certainly, make the most of your holiday - and it would be lovely if your brother is able to spend a bit of time with you but, that will depend on so much else.
You are not a bridesmaid.