I think there’s upsides and downsides to AI…for example AI is capable of making these images exist BUT AI is simultaneously capable of detecting inappropriate and illegal images. I feel like it can therefore be possible to not only prevent AI from producing content like this, but to automatically flag and report it if someone is prompting illegal images to be produced. It’s just down to whoever trains the AI model to build in these flags manually to place limitations/safeguards in.
The company in question behind the software should also hang their head in shame really. It’s a big failing on their part. With ChatGPT for example you have to brute force AI to get to behave in an unintentional manner (such as swearing at you) and even then those exploits are regularly patched. Let alone getting AI to behave in an illegal manner such as this criminal.