oh come on.
You bring your children up bilingual if that is what you want to do. Your kids, your choice. You and your husband agree on that? So that is what happens, your in laws have no say.
When everyone is present, you speak to your children in your language and your in laws in English. That is not rude, it is the established way of teaching children both parents’ languages.
If your kids are having a conversation in English with their grandparents and then you join the conversation, you join in in English. I can see that big family meals where everyone speaks one common language (in this case English), the default should be English for the most part. Why would you want to actively exclude people from a conversation when you don’t have to?
The majority of the time, you are speaking to your kids in your mother tongue. Believe me, it is enough for them to be totally fluent, particularly if they also have wider family that speak it with them exclusively (your mother etc).
Don’t be a drama llama and make a big thing out of it. Be polite and reasonable.
As an aside, talking to your mother in your own tongue in front of your in-laws (and about them as well!) is breathtakingly rude. I can absolutely see why they called you out on that, it is appallingly bad manners.