It is quite a fringe issue in most peoples’ lives
Quite possibly true. Which is why there's not been more pushback yet - a lot of people still have yet to encounter its effects, and are living in blissful ignorance.
The level of pushback you're seeing now is what you get when a huge chunk of the population think this isn't really happening, cos it hasn't hit them yet.
People suddenly wake up when they find a man in their single-sex rape crisis group, or sport, or any other single-sex space. Or find their child deciding they have to have puberty blockers.
Whatever anti-gender-identity-ideology pushback you're seeing now can easily be many, many times larger. We know there's a huge mass of unaware people, and we know which way most will go.
Everything we do is about sunlight - getting people to listen to trans activists, and watch what they're doing. We don't really have to make arguments - just raising awareness of what's happening is sufficient.