I don't disagree with you. But a BTL intentional landlords with multiple properties is very different to 'accidental landlords' who rent their property short term due to a specific need (in my case I had to move for work and couldn't sell my flat due to leasehold issues - would have much preferred to just sell).
However, given this government is neck deep in debt and is unlikely to be able to afford to build enough social housing for all that need, and not everyone wants to or can afford to buy, where will people live if private rentals dissapear? As you say, many have been pushed out as it doesn't make much money anymore, and the consequence has been rents being driven up, and dodgy landlords thriving off desperate people. The number of registered landlords is down yes, but the number of unregistered 'beds in sheds' has more than doubled in the London borough where DH works in housing. And that's only the ones which come to their attention due to health and safety, pest control issues etc. And many of these people would never be eligible for social housing anyway due to immigration status, length or residency anyway.
Make it harder to do something legally, people will just end up doing it illegally. Whether thats drugs, abortions, or housing. All that happens is dodgy people thrive and the vunerable become even more so, and at greater risk of harm when forced outside of an at least semi regulated system by price or lack of availability or ineligibility.
I am very pro social housing for those in need, I spend a good chunk of time trying to get my patients further up wait lists. But there will never be enough, especially in built up areas in London.