I've never heard of the tv license 'inspectors', but something similar happened to me a couple years ago with the electricity metre reading.
A man turned up, tried to open my door and then banged on the glass so hard I thought he was going to break it. I went to the door, he said he was there to read the metre, I said it wasn't a good time and he tried to shove the door open and push past me. I managed to get the door closed and he screamed through the door for 20 mins before leaving. He had quickly flashed a company id at me when he was trying to force his way in.
I phoned the electric company to complain. They told me they hadn't sent anyone to my house and told me he was 'clearly trying to break into your home' and advised me to phone the police. Later that day, the man showed up again!! Again banged on the glass so hard I thought he was going to break it, yelling that I had to let him in, I was breaking the law, etc. I told him through the door (didn't open it) that I was phoning the police and had already phoned the electric company who told me he didn't work for them. He screamed abused and eventually left.
He showed up at my house every day, several times per day for just over 2 weeks! I didn't open the door to him, but he would threaten to break in, screamed abuse, 'I know you're in there', etc. I phoned the police (as did my neighbours) and I kept reporting to my electric company and questioning 'Are you sure you haven't sent this guy?? He keeps showing up and claiming he works for you!' they kept 'assuring' me he didn't work for them, they hadn't sent anyone to my home and they had my current readings because I was reporting them online (with photos as proof). I was speaking to senior management at that point and even they were telling me to report this guy to the police.
Eventually it all came out that this guy did not work for the electric company (like he said he did). He actually worked for the company the electric company had hired to do readings. The hired company said they were being told to come to my house, the electric company was saying they never told them that.
Needless to say, we switched electric companies. But I do agree with a pp, these contracted companies hire thugs and it's scary to think they're given your private details (name, address, phone number, etc.) and let loose to behave in that way towards unsuspecting customers.