It's been a long while since I've worked with RNA and I've never had to prepare it from actual clinical samples, but I can offer some possible ideas of why the current tests have a high incidence of false negatives.
(1) Depends how much viral material is present in the patient's airway at the time of swabbing.
(2) I do not know how the swab is prepared for analysis but RNA is very vulnerable to breakdown so could be lost during the process due to RNAse enzymes secreted from [well, every living thing including people's skin] and sticking around on surfaces.
(3) Further to that, the RNA could again potentially be broken down (for the same reasons) whilst the RT-PCR itself is being set-up.
When I worked briefly on RNA analysis of laboratory samples, we kept aside a particular laboratory bench which had to be freshly and specially cleaned down prior to use, and kept separate special RNAse-free equipment, plasticware, and chemicals. And this was using the best industry-standard kits of the time. To be honest, I really hated the hassle and was delighted to abandon those experiments at the earliest opportunity 
(Like I said at the start - it's been a good few years, I imagine that the protocols, technology and kits are now much improved with regard to protecting the RNA during sample preparation).