No it wouldn't be 'an entirely different story'. He would still be a man in a women's prison where he did not belong.
It's not enough to say he probably won't physically attack female inmates. It ignores the psychological harm and distress and the loss of dignity for female inmates who are made to share with males.
And the vast majority of sexually violent men are never even arrested, let alone charged, let alone convicted, let alone imprisoned. Just because he got caught for fraud or tax evasion that does not mean he has not committed violence against women or would not do so given the opportunities and stresses of prison.
After all, one of the reasons we have single sex spaces in the first place is because we can't tell which men are dangerous and which are not so we keep them all out.