Exactly.
The Queen retained her popularlity by never expressing her views and seeming to put duty above any personal gratification. Monarchists could tell themselves that the Queen's views aligned with their views. However, there were periods during the late Queen's reign where the popularity of the monarchy dropped quite significantly, particularly after the death of Diana.
Other members of the Royal Family have not been as discreet and circumspect as the late Queen. I believe that Princess Margaret's and the Queen Mother's views were very problematic regarding race and class, and the Queen's uncle was an unapologetic Nazi.
The point is that, no matter how bad their behaviour, we cannot get rid of the monarchy like we could get rid of an elected President. I believe that Andrew was always the Queen's favourite child, even after all the sexual misconduct allegations and the paying of £12 million in hush money. This doesn't show her judgement in a good light.