Where I live, second homes have killed some of the local village schools, shops and pubs, because when the properties are empty for most of the year, there is not enough people/trade to keep everything going.
One local property used to employ 4-5 people, pay tax etc, and now is lived in for a few weeks a year.
At least if a property is rented out privately, there could still be a family living in there, contributing to local costs, or even a holiday home rented out for a lot of the year keeps the shop and pub open (but not the school).
I do think that there should be specific areas/villages where it is banned and that you should have to occupy the premises all year round to be able to buy it. There are plenty of properties out there, and lots of building going on, where people could have bought beachside/city apartments for second homes, that would not have impacted on daily life in villages.
In some of our local villages, there are people who have moved there from away, they do live there all year, but are older retired people as the young couples can't afford the property. These same people then oppose all new affordable housing building, and try to stop young families moving into the area. It is so unfair and most people can't afford to live where they grew up.