The 2004 GRA was designed specifically for transexual people with the medical condition Gender Dysphoria.
Here is a quote (from Hansard) from the original debate in the Lords, 2004:
we are talking about only 5,000 people in Britain. That is why this is such a unique and specific set of circumstances. This is one of the smallest minorities in our society
When the law was created in 2004, everybody was clear transvestites, cross dressers, men with sexual fetishes etc were not the same as transexuals with gender dysphoria.
Trans now means transgender not just trans^sexual* and this much larger group does include people we previously called transvestites & fetishists, along with people transitioning for a variety of reasons other than medically diagnosable gender dysphoria.
This law was never meant for those other groups, and government has not debated what the effect of that wider group having access to GRCs would be, nor what the effect would be of hundreds of thousands of people, potentially, getting a GRC would be on society, rather than the originally intended 5,000. (The GRA consultation is using an estimate of 200,000 - 500,000 trans & NB people in society).
It's disingenuous to frame this as a simple administrative change when the goalposts have been moved so far as to make this a different game entirely.