You don't need to know what's in people's underwear to pour them a pint, buy a used car from them, arrange a mortgage for them, share a table with them in a crowded cafe, have your letters delivered by them, etc etc etc.
Apart from when you're contemplating sex (or TTC) with someone, or you are a medical professional needing to treat an ailment, are there many other occasions when it actually matters what sex/gender another person is?