As a few people have attested on here, it's sad to see that disabled people are often just brushed away and othered, when told they should seek different types of accommodation which are completely unsuitable for their needs. Should disabled folk be 'punished' for their disabilities by being told that holidays are just not for them, along with all of the other things in life they can't easily do, or do at all?
We always have the same othering mindsets when people scream for car-free city centres, with no private vehicles allowed at all. Proclamations that clearly excluded any other minority group in society would be roundly condemned, but 'ablephobia' seems to be widely acceptable
Of course, it's not just the most severely disabled people, but many others also have physical/mobility/neurodiverse/cognitive issues - as well as many elderly folk - but the younger, able-bodied, healthy people often seem not to consider them as actual people who actually matter 
I presume that the hatred for holiday lets, taking away a house that local people could have lived in, also extends to people who live in houses that are significantly bigger than they strictly need? Under-occupation has a big impact on many communities - not just touristy ones - but nobody seems to have anything like as much beef about that.
Taken to extremes, we have the situation where a couple living in a tiny house/flat in a city as their main home and owning another tiny house/flat near the coast that they visit when they can are considered evil; but another very wealthy couple who live alone in a massive 8-bedroom house in one location are completely uncriticised - an 8-bed home that could accommodate two or three families if reconfigured or replaced by several modest homes.
Of course, nobody likes it when streets and roads are blocked and it's hard to move around, but this is not just a holiday thing. I live in the Midlands, where we have a number of major motorways, and certain of these are also regularly gridlocked. I'm guessing that many of the users are not based in the Midlands themselves, but if there's a public place that attracts people - whether because of large trunk roads, historical landmarks and museums, beautiful towns and villages, the seaside or whatever - you'd have to be dim not to realise that the crowds will come. As an alternative, you could always live in Jaywick or Rotherham if you don't want floods of incomers. People always complain about congestion, without appreciating that they are just as much a part of the problem!
Something else that strikes me - and I'm playing devil's advocate here, as I don't really believe it myself - but if we go from a standpoint that anybody with enough money who can find a willing seller is able to buy a house, five houses, twenty houses in a beautiful place, maybe that gives them the right to then use it as little or much as they like, just the same as somebody who only has one house there. Maybe their interpretation of what they want the community where they own to be is like a less artificial Disney World - somewhere set aside for holidays and leisure time, where workers commute in to make it function.
As I said, I think it's terribly sad when this happens - but does one stakeholder (houseowner) who wants the place to be one thing automatically trump another who wants it to be something very different?
A rather vague analogy might be something like Stonehenge, which is run on the basis of incoming tourists and their desires, often to the detriment of Druid and Pagan worshipers who then often find themselves shut out and unable to get close to the stones which are holy to them. How do we reconcile that: the overwhelming majority of general sightseers wanting to see a spectacle against the minority of folk who hold the stones as an extremely sacred part of who they are?
I also think it's a bit too simplistic to assume that everybody who lives locally is good for the local economy and all non-locals are bad. Yes, locals will keep the GPs and the schools open, but when it comes to shopping, are we to believe that all locals extensively support local shops - even when out at work all day - instead of buying most of their stuff online, as so many people do nowadays; when compared with people on holiday/leisure time and as such with extra spending money and the freedom to wander around local shops?