I agree if you have more than 10 cars piled up behind, then you should dip in and let them pass.
However, not everyone is sure of the edges being hard, or that they won't be stuck off road for ages waiting while there's a safe spot to pull out in.
Guilty: in my yoof, occasionally I've driven like a rally driver along rural roads I knew and slowed down to an almost stop in places where I knew were tricky- think barren Heath lands (no hedges) with excellent visibility up road, but with dips and bends.
However, I agree, where there is poor visibility, unclipped hedges, and tractors, mowers, horse boxes, horses, dairy cow herds etc, it makes sense to keep it slow and stick into the left, even if that means cars are piling up behind.
However, if there were tourists in hired cars, driving on their "wrong" side, I'd be patient, and let them drive the way they felt comfortable with. I'd just get there later than I intended, rather than try and pass. Sometimes I'd stop and whip out my thermos, and have a cuppa! They'd be gone ahead enough when I'd finished.
There are other considerations to consider when driving and that is noise pollution, and its effect on the communities of animals living there. Slow speeds are sometimes there to ensure vulnerable animals and birds can life their lives without being stressed from the noise of engines and wheels.
So yes in a way I agree with you OP.
If cars are piling up behind in a queue then it's polite (and Highway Code) to pull in and let them off, but if there is a tourist or someone who isn't sure of the road then I'm afraid slow and safe is better than driving like hell and getting there.
Sometimes you've just got to suck it up. At least you've fresh air and nice scenery, eh?