I don't think they can access your property to check whether you use a TV? Or is that a myth?
They have no right of entry to your property whatsoever. A knock at your door is nothing more than a polite request that you answer, and you are under no obligation to do so. If you do open the door and find one of the goons standing there, you are also under no obligation whatsoever to interact with them in any way. You can simply close the door on them, or if you are feeling slightly more charitable, a polite 'no thank you' and then close the door.
The only time they can compel you to permit entry is if they turn up with a warrant from a local sheriff or magistrate. They will be accompanied by police in that instance, however, they will not be able to attain a warrant by simply approaching the court on the suspicion that you may be watching TV without a licence because you closed the door on them. There would have to be substantial evidence before a court would consider granting a warrant, so if they have perhaps witnessed you watching TV through a window while approaching your door, and then you refused to engage with them, or you invited them in, they saw a TV on, and then you flung them out the door.
Quite simply, the letters are circulars that they send to any address without an active licence. They are not personal, they are not printed off because of 'suspicions', and the 'active case' stuff is purely a scare tactic. Their enforcement officers are employees of Capita. They have no authority or powers beyond that of any other random member of the public. They can not compel you to engage, and if they try the nonsense of 'reading you your rights', which they have been known to do, try not to laugh, and just close the door on them.
The people who end up being fined are invariably those who naively engaged, and then incriminated themselves, either through admissions, or by permitting entry to their property whereupon a TV was witnessed to be in use by the Capita officer. If you simply ignore any and all communications from them, and refuse to engage in any way, there is not a damned thing they can do, because they have absolutely no powers of their own.