I think this is true of any position. There will be people coming to it from all sorts of angles and holding all sorts of other positions alongside it. You aren’t ever going to agree with them on everything. And sometimes you will find that your views on one thing align with the views of people whose other views you find abhorrent.
I actually find the expectation that people should holistically hold the same views really troubling. You often see this used to dismiss a perfectly well made, reasonable point: oh but we can’t listen to that because that person also votes Tory/supports abortion/once said something racist/whatever else you’ve decided is unacceptable. If you see it in threads asking what ‘our’ position/argument is about X.
It’s always going to be a diverse coalition of people whose views align on particular topics. And that needs to be OK. The fact that people from across a broad range of spectra (on all sorts of issues) coalesce around an idea doesn’t invalidate that idea. It is actually pretty compelling evidence that the idea has wide support across society.
It’s likely that many white supremacists’s views on chocolate will align with mine, for example. That doesn’t make likening chocolate a bad or shameful thing. The chocolate isn’t tainted because people whose other views I find dreadful also like it. Being gender critical is a much less trivial example, but it is the case that my views on it are going to align the views of individuals or groups with a range of other views.
critical thinking about the peculiarities of specific issues is really important. Throwing the baby out with the bath water is not generally conducive to critical thinking.