Yes, as PP said, people of a certain generation (mine, sadly) used it as an insult. Came from a Blue Peter appeal to build a home for Joey Deacon, a man with cerebral palsy.
There's also the church associations, and it makes me think of Deacon Blue (a band that were around in the late 80s).