Any attempt to guilt me into it is manipulative
I don't think it needs to be about guilt and you are free to have whatever beliefs you like. As are we all.
Brain tissue is sex steroid sensitive. That is factually correct. It is likely that prenatal hormones have an influence on brain development. That could influence gender identity development along with other factors including environment. The mechanisms governing aetiology are feasible.
Even if it there is no biological component (and most medics/ researchers believe there is) it doesn't invalidate the concept; cross sex gender identity is something that has been observed in clinical settings for many decades.
If society comes to one set of understandings, laws and processes and you are opposed to them then that is your right. If the greater part of society legitimises those standards and ways of being as both custom and law, then of course you have the option to opt out of at least one of those, but not really the other. You can say what you like in your own home, but you can't misgender someone in the workplace and expect to keep your job.
I realise this isn't always the case, if the suffragettes hadn't gone against society and the law then women wouldn't have the vote. But I'm assuming human civilisation is in a different place today and that society is making more rational decisions than it did 100 years ago.
If you see trans women as men and feel that you are being forced to accept them in female space I can see how that could feel like having your boundaries violated as has happened to you before. I feel like that is maybe painful for you (could be wrong) and making you angry. I don't think that is trivial or unreasonable. I don't know what the solution is.