Apologies; long post. I have been thinking about this a lot. I would say there are different reasons, depending on audience.
The victim narrative, drawing on genuinely distressing stats from South America, has been coopted in the U.K. and USA. People don't check stats, as a rule.
Also, the general public has been led to believe that there genuinely is a 'hate mob' who oppose the human rights of transpeople. The ones I have discussed the matter with genuinely have no idea that the 'hate speech' (from what I have been able to see from my admittedly limited vantage point) consists mainly of disagreement with the position that pre-op transwomen people are, in the ontological sense, female based on their say-so. Once people understand that, they don't understand why this would ever be seen as 'hate speech'. They are incredulous, and find it hard to believe that's all there is.
Those who do know about this and still don't see why there may be resistance to self-ID generally don't realise that it's a matter of conflicting rights under law.
Part of the problem is that trans activist are operating in quite a strategic manner, e.g. the Equality Act does allow for exceptions on a case-by-case basis for putting protection of gender ID at the same level as sex, which some quite reasonable legal people point to, without realising that there has been a grassroot-level campaign to get individual councils to remove this exception by switching 'sex' for 'gender' in local regs. The two campaigns have seemingly taken place in tandem, but on different levels. One would have to engage closer than most without 'skin in the game' tend to do to see the connection.
Re. the 'hate speech' argument, the TRA discourse that cannot possibly be filed under the 'victim' category, e.g. the calls for rape, assault or even murder of so-called 'T*RFs', does not take place in full view, but is directed at individuals on social media or circulates in LGBT+ or feminist media sources. The general public simply don't see it. Those who do are horrified.
Basically, mosr people are quite nice, and are willing to believe a victim narrative and give people the benefit of the doubt.
Is there more scepticism towards women than men, i.e. is there a sense of underlying misogyny? Possibly. Arguably, someone who subscribes to the feminine norm - 'feminine' in the cosmetically aspirational way, and tending towards the submissive, long-suffering victim role, will be seen in a kinder light than the stereotypical 'angry woman' (especially if they can be imagined as 'middle aged', 'old', 'ugly', 'hairy-legged' and so forth - whether that is the case or not). It is very easy to dismiss women as 'screeching harridans' - again, whether this is the case or not.
Let's also not forget that the public generally also relates differently to suffering or victimhood that is common, seen as depressingly inevitable, and beyond what is 'solvable' with some kind of 'fix', compared with something that has news value, and seen as 'fixable'. An element of titillation also helps. The interests of vulnerable women in refuges or prisons will have a hard time 'winning' over reasonably glamorous transwomen in the media. It is depressing to think about the former; fun and cheering to think about furthering the rights of the latter. We tend to avoid engaging with genuinely difficult problems to 'solve', like domestic abuse.
Funding is possibly also a factor. The lesbian community, who currently seem to be in receipt of the sticky end of this, receive something in the order of £4m per annum, according the figures I saw. The corresponding number for transpeople is £22m.
All these things likely matter, and a coherent response hinges on joining the dots. This takes interest and engagement, as well as media exposure.