I really don't think there is entrenched ill will.
People have a soft spot for PH (right back to the funeral parade) and have been ready to forgive him appalling lapses in their youngerbdays, and if he's happy, they'll be happy. No one really cared much about their costs when they were here as working Royals, and weren't expecting a huge workload of them when children were small - their six-week break passed unnoticed until the very end of it and I don't remember anyone begrudging them it.
It is the level of support whilst they are not working Royals that is likely to lead to questions about public and royal finances.
Non-working royals are meant to pay a fair(ish) rent for accommodation. £18000 is low/typical for property that size (in terms of bedrooms), but in less desirable nearby location and (probably) without a number of the features. Which means they are doing very little of the paying back of the costs as they have agreed to do.
That's a bit of an issue whether we love them or not.