I'm referred to as his wife, in any situation where we're together as a family. It's not exclusionary, it affects me not at all when this happens
That's great for you, but asking a child where his mum or dad is does make a child who has two mums, or might be gay themselves feel like an other in a heteronormative society.
Assuming that everyone is in a heterosexual relationship and using language to suggest that this is the norm makes people who fall outside of those categories feel excluded and different.
It's the same for kids in single parent families - people asking where their mum or dad is can make children feel there's some deficiency in being raised by a lone parent, but 'parent' is neutral.