The magic word is 'minority'.
It is accepted that minorities need specific, protected rights in order to have true equality with the majority. It is accepted that only minorities can define themselves and define what oppression and prejudice are for them - these cannot be done by the majority.
People with a strong ethical and moral compass will automatically support advancing the rights of a minority, especially one that clearly defines its oppression and presents solutions.
Are the people who think women's sex-based rights should be given to males equating the demand of trans ideology with, say, the removal of quotas of Jews at university, or allowing non-Christians to swear oaths appropriate to their faith on their holy text, or banning signs stating 'no blacks' in the windows of boarding houses? Generally speaking, granting rights to a minority does need harm to the majority - just makes them share. So there is more competition for a place in university/a seat as an MP/a basic room to live in.
Do people think that, by definition, any minority must be given the rights they want? That it's unreasonable to refuse to share with a minority?