Basically this:
In the late 90s, someone that David Jason knew was convicted of child abuse. It transpired that this person had been asked by Jason to keep an eye on his house when out of the country (in Australia) and he took the opportunity to commit one of his crimes there. At the time the friend and another man were convicted of many abuses going back years, the Police made a statement that their investigation showed Jason had no knowledge and was indeed overseas.
You can find the assorted press reports online.
So basically, the rumours come solely from that and "guilt by association".