2022 US Transgender Survey (Interim report Feb 2024) - looking at methodology

[Gettingmadderallthetime]

This was reported on CNN. (I found it by looking at posts on X that disagreed with some of the posts I agree with - trying to understand why people are disagreeing and what evidence they use). Putting this here because we may see this survey referred to a lot as its very large.

The eyecatching info is 'The 2022 US Transgender Survey Early Insights report, conducted by the National Center for Transgender Equality, polled an “unprecedented” 92,329 binary and nonbinary transgender people ages 16 and older living in the US, its territories or military bases, according to the report.' and a very large minority were very happy with their transition. For example: 'Ninety-four percent of respondents who lived at least some of the time in a different gender than the one they were assigned at birth reported being “a lot more satisfied” or “a little more satisfied” with their life, with 79% expressing the highest level of satisfaction, according to the report.'

So I looked for the report and at the methodology. There will be more about the methodology in the full report (yet to be published). So should I trust this as a source? Looking at what is said about the methodology ...

• You could only take this survey via the USTS website. 'The survey was hosted by Qualtrics and could be accessed exclusively through the USTS website (USTransSurvey.org).'
• There were 605 possible questions across 38 sections. Respondants were routed through these so no respondant received all questions.
• There are NO NUMBERS given in this interim report - just percentages. It is not explained what the percentages refer to but given the above points this 94% cannot have been 94% of the survey population of 92,329. Its 94% of whoever answered that question having been routed there.
• It took 60 mins to take the survey. There were prizes offered but these won't have been a significant incentive. 60 mins is a long time.
• USTS encouraged pledges. 'the team conducted a survey pledge campaign, which was among the most important methods for engaging and communicating with potential respondents. The campaign invited potential participants and allies to pledge to take the survey and/or spread the word about the survey. The survey pledge was designed to raise awareness about the survey and engage potential respondents for a sustained period leading up to the survey launch. Individuals who completed pledge information received email and text communications throughout the outreach period. The pledge was an important component of the outreach and communications strategy in the 2015 USTS, and the large number of pledgers in 2015 (~14,000) was thought to correspond to the eventual large number of respondents (27,715). The 2022 USTS outreach team improved upon the survey pledge campaign to substantially increase connections and engagement, resulting in 34,576 people who pledged to take the survey before it launched, 12,015 of whom also pledged to share the survey with other transgender people in their life
• The communications team created promotional materials and messaging to share through email, social media, and other methods. They maintained communication with thousands of individuals and organizations, including people who pledged to take or spread the word about the survey, organizations that committed to support the survey through outreach efforts, and people who had signed up to be in communication with NCTE about the organization’s work and projects more generally. They also developed a “partner toolkit” with materials for organizations to download and use, including key messaging, promotional graphics, video scripts, social media posts, event materials, and language for emails. The team provided information through many channels, resulting in the survey being promoted by influencers, organizations, and content creators through social media platforms, such as Instagram, Twitter, Facebook, TikTok, and Tumblr. The team also commissioned videos from key influencers to promote the survey prior to the survey launch and during the data-collection period, including “progress videos” that were embedded in the survey to thank respondents and encourage them to continue completing the survey. In addition to providing materials about the survey, the USTS team held dozens of events to raise awareness about the survey, such as Instagram and Facebook Live events to discuss the survey with influencers and organizations.
• The outreach team worked with organizations to reduce barriers to accessing the survey and increase opportunities to take the survey for people who may otherwise not have had access. One such method was by working with organizations to organize “survey-taking events.” These were events during which organizations provided a location and resources for attendees to take the survey, such as computers or tablets. These events were intended to provide access to individuals with limited or no computer or internet access, those who may have needed assistance when completing the survey, or those who needed a safe place to take the survey. The team also ran a tablet-loan program to provide another avenue through which organizations could offer survey access.
• Noting that the survey is anonymous I wonder whether they are cleaning data to remove any duplicate/multiple responses.

They complete the methodology by saying: 'When interpreting the preliminary findings presented in this report, it is important to note that although the
team sought to recruit a sample that was as representative as possible of transgender people in the U.S. and analytic weights reduce sample biases, study respondents were not drawn from a random sample. Therefore,
while this sample is a large one, the findings may not be representative of all transgender people'. I am not clear from my points above that this is representative, still less 'representative as possible'.

The final version of this will be an interesting report because it seems clear to me that they reached pretty well every pro trans person in the US that they reasonably could have. They could not have thrown more resources at doing this. It may provide a very powerful insight into the world view of the trans community in a #letthemspeak way. Its not representative of views/experiences of anyone who is not very pro trans and pro USTS.

Oh and the Guardian give it coverage. https://www.theguardian.com/us-news/2024/feb/07/trans-survey-transition-gender-affirming-care

https://transequality.org/sites/default/files/2024-02/2022%20USTS%20Early%20Insights%20Report_FINAL.pdf

Great work on the methodology, Gettingmadderallthetime! Can we run every dodgy piece of research past you please?☺

The bit that stood out to me was
Ninety-four percent of respondents who lived at least some of the time in a different gender than the one they were assigned at birth reported being “a lot more satisfied” or “a little more satisfied” with their life, with 79% expressing the highest level of satisfaction, according to the report.'

Well if they'd asked me I'd say I'd spent pretty much all my life 'in a gender different from the one I was assigned at birth' otherwise I wouldn't have had the clothes I've had, the jobs I've had, the girlfriends I've had, the hobbies I've had, the haircuts I've had, and I would have expressed 'the highest level of satisfaction' without having to pretend I wasn't a woman.

If we imagine that it was a survey of gay people rather than transgender people, I think you would be less concerned about the methodology. I suppose it might end up with disproportionate number of happier gay people. It would still provide useful information about the attitudes and experiences of gay people, though not gay people who are anti-gay, or don't consider it an important self-defining characteristic.

This survey is not designed to look at the experiences of detransitioners, or find out their numbers. It looks at people who are transgender, and their experiences.

@DadJoke it could be a survey of women who knit for all I care. The methodology is unhelpful in telling us anything about how trans people in the general population feel, based on being

• Coached (social influencers and others did training about this survey before it ever was launched).
• People pledged to bring their friends (people with similar views I assume) into the survey.
• Supported at events and through loans of equipment and only accessible through a pro trans site.

It will give us insight I feel into the group that it carefully selected to survey. As long as they can sort the double counting from anonymized answers and not sure they actually can. That in itself is interesting. How are people who are very much TWAW and uncritical of trans affirming care etc. seeing the world. I am interesting in learning more as I don't understand and would like to. (This is why I am looking at the evidence that is being put up against GC evidence. I look closely at GC evidence also BTW).

This is an interim report. My take away is that for this group who were selected to be uncritical of the survey (it feels) they were indeed uncritical of the survey. So what are their views on other things.

(The methodology states that this was not a random group (what they are saying is that even within the trans population this is not a random group). They do not give numbers. That is a red flag when there is routing around questions and 600+ questions used. Its a very bad survey design but hoping there is something useful as its a lot of effort. And money.

@MarieDeGournay thank you but no. I am not a methodological whizz but because I don't find it easy I really had to force myself to learn what is good and bad research design for my PhD. So I pay close attention to methodology sections.

This one is appallingly bad. They could have afforded good research. Two years to get to interim report given the resources on conducting the survey.

@Gettingmadderallthetime if you wanted to do a huge survey of gay people, how would you go about it? You are framing this as pro-trans and anti-trans, but you wouldn't be using the same framing (gay vs anti-gay) if it was talking about gay people. It's peculiar, and being pro-or anti-trans is not something you'd expect to find in any sample of transgender people in large numbers.

Self-selecting surveys are not great methodology, but with such a large sample set, I think we'll get some useful data. I suspect it will overestimate their happiness and underestimate the amount of abuse they get. Poor trans women of colour, whose experiences of being trans will be very different, are much less likely to be reached.

However, they can use indirect indicator to adjust this - for example, educational achievement.

The survey might have bad questions, it might have good ones, but it's weird take to suggest that people might be critical of the survey and therefore not participate. What questions might transgender people be critical of?

The survey definitely appears to have flawed methodology. Its hard to tell as it conceals more than it reveals at this stage. The problem of having good questions amongst bad ones is, if the methodology is flawed, then there is no good data to build on. If you feel that TWAW and should be recorded in crime statistics as women then you know what I am talking about. There may be good questions, but there are undeniable doubts about what the answers mean.