And you are entitled to your opinions on the interpretation of what he said.
I, however, don't feel that you can separate out what he said to be aimed at politicians and only politicians. And besides which, which politicians? There are politicians who are discussing the very same things that feminists are discussing. That you, personally, feel that they are just indulging in distraction and should not be trying to tighten laws or to get out some clear guidance on this issue is your own issue to deal with.
People with CAIS are, in fact, male people. And it is yet to be evidenced that for the purpose of sport that they do not have advantages over female athletes and should not therefore have their own category in the future. It is not a 'culture war' to discuss this.
Also, a government needs to be able to have many focuses. So, again, framing this as being not important is your personal opinion and there are plenty of others who disagree with you about the level of importance.
However, this thread is about what he said. Regardless of what you believe the priorities for government debates and discussion should be, he has certainly made the statement he made. If he was being specific, then he should have clarified. Did he? Has he? He has had time to do so.