Strongly suggest you contact your MP, and adult social services and object hard. Given it's been smashed and the claim is ridiculous, you are entitled to take the line this may in fact be verging on a hate 'crime' that the ASBU is now assisting.
There is either more to this and the issue isn't the actual key safe - such as carers on their phones loudly at the wrong times of day &night?
Or Mr ASBU has let the job go to his head/ stopped evaluating before leaping to action, when faced with an unhappy resident complaining.
When I first came out of hospital after a long stay neighbors were up in arms that it wasn't acceptable to have a stream of carers able to access a key to open their (our) front door to access my flat door, and I had 'too many visitors coming and going' as there where also nurses, frequent phlebotomists, and secure drugs deliveries. They were also unhappy that they wouldn't answer demands for information about me to them, and it was them being shut down that caused an escalation of their complaints.
I'd have more sympathy for them feeling insecure, except they were buzzing in all sorts of delivery folk, meter readers, kids mates, and whatever dealers their older kids had found online day and night, and are large families thundering up and down the stairs with no normal hours. I hear most of what they say at all times.
Suddenly they could hear people talking to me, and my existence and presence was no longer invisible to them, and other people were witnessing what they felt to be their private lives. Ie all their stuff in the communal areas and who they had visiting. They are hugely territorial and expect to never be affected by living in places with communal spaces and doors, and will pick up their mail from the floor but leave mine on the floor and trample and kick it around, instead of putting everyone's mail on the side to be claimed like normal people do.
As long as I'm silent and invisible I'm not an issue to them, but they feel I have less 'rights' than them because ablest hierarchy tells them that.
Check what is actually going on here and that your carers aren't being unacceptably loud.
Don't allow yourself to bullied just because your neighbors 'hijacked' an idiot in authority. Challenge it.