YABU, of course. I used to think that it was discriminatory, but then rented my mum's house out to a mum on HB - good references, family known to neighbours, little boy just about to start school so thought she'd want to have a stable home.
Four months into the let, she stopped paying rent, hid from agent, refused to answer phone. We gave her several chances to pay, tried to agree payment schedule for arrears, still no payment. Then heard that she'd been slagging my mum off to all the neighbours, saying that she was being kicked out because we'd sold the house over her head.
So shortly after that, we gave notice for her eviction. We're now several hundred pounds down, and this is money that funds my mum's fees for a (not luxurious) care home, so not something we can afford to lose - we don't have the option not to pay care home fees.
I'd still rent to tenants on HB, but only working people, because they've got more to lose if they piss you about. I'd never rent to anyone without a job now, HB or no HB.