I guess it depends on the purpose of the conversation as to whether you are being unreasonable or actually have a point.
My perspective is that talking about money in an objective fashion isn't crass but then, to me, it is nothing more than a matter of fact and has no bearing on a person or how I see them. That said, I am aware many people are either uncomfortable talking about money or think it is crass to do so, while others might need to have a conversation because they are struggling and they are just not in a place to hear how wonderful someone else's financial situation is.
Think of it this way... If your relationship with your husband was absolutely flying, and a friend was going through a divorce, you wouldn't start talking about how wonderful things are in your life while they're leaning on you to or trying to share their woes. I can't imagine there would be an awkward silence, either.
If a friend is talking to you about their financial woes, listen. Try to see the world through their eyes. Offer sympathy. If you have any ideas on something that might help, make sensitive suggestions. If they're really down on their knees, do something for them - send them a hamper with a note to say you know they're having a hard time and want to cheer them up or take them out for a meal just because you love them and want to give them a break.
If the talk is just general "jeez, butter costs more than my kidney these days"... Like, yahhh. Your financial planning hasn't somehow paused the crazy cost of living increases for you and, whether you can afford them or not, surely with such financial acumen you've noticed the numbers rising at a crazy rate. I'd also hope your maths skills are relatively en pointe and you must be able to figure out that x + y no goes into z. That's nothing to do with your own financial position so I'm not sure why it would be awkward to talk about it.
If you want to have a proper, in-depth talk about finances, which is an absolutely fine thing to do, just choose your audience carefully. You wouldn't strike up a conversation about your wonderful marriage with the friend going through a divorce, would you? Similarly, if you wanted to talk about your sex life, you'll have friends who'd be happy to engage in that conversation and others who would be extremely uncomfortable to do so. If you were swingers, I imagine you'd only talk about swingers stuff with fellow swingers. Same with discussing your investments.
Ultimately, I am concerned that you place too much importance on money - as though it makes you different to your friends because you seem unable to separate yourself from your net worth (congratulations on that, by the way, you're going to have a very comfortable retirement and I appreciate that will have come with hard work, going without and taking a sensible approach to money). The idea that you don't buy things for the reason you don't want to stand out is a bit weird - I'm not sure anyone really cares what you drive, do they?! If I had the money, I'd have that Lambo and I'd be taking my friends out in taking, chauffeuring their kids to prom. Our multi-millionaire friend used to have supercars and would drive one while basically lending my husband the other so they could "go out to play"!
I've been a poor student, hanging out on a yacht with the multi-millionaire mate (all expenses paid), I've had friends take me out for meals because I was working four jobs just to pay my way through law and not a penny to spare, send me little gifts to cheer me up. They didn't judge me, they didn't see me any differently to them (just a friend in a squeeze) and I didn't judge them (I didn't think they were better than me, just good friends looking out for me and loving on me). I was grateful for their kindness and knew I'd do exactly the same if I was in their shoes.
I've also been financially stable, taken friends out for meals, bunged them some cash etc when they've been in a bind and explained that others have done it for me, before - if it makes them feel better, they can either return the favour when they're in a better position or pay it forward, but they don't need to.
I now live in the "forever home" and it's really fucking awesome. Guest rooms are always made up ready for friends, and the door is open for anyone to come and eat with us or just hang out here. Our friends know they can come and go as they please, and we're obviously in a position to be able to look out for them. It doesn't cause any issues. In fact, we've had friends use our house to host parties just because it's big enough to hold everyone and meant that the group (which is now a bit spread out) could all be together.
Just don't be a dick about it, I guess.