The British public are very forgiving of the royals
I disagree. Some of them might be, but the media are not, and they have the main away of public opinion. Look at prince andrew, his stepping down was an unprecedented move made due to the pressure of public disapproval.
If they returned the press would have an absolute field day, they would continue to be pariahs among the other royals and they wont have friends to turn to as allegedly they lost them all.