I was beginning to think it might be our cat, until you mentioned the fact he was affectionate.
We got him when he was supposedly 16 weeks old, and from a woman who it became increasingly apparent, was a bit of a loon (telling us we were not to look at her cats or the or the kitten we had directly and other random stuff).
Anyway it turned out he was older than she said, and had socialising issues, as he had been isolated from any human contact aside from the mad cat lady, and the vet was unsure if he would ever fully overcome this. As well as that, we were told he was fiv positive, so decided the best thing would be to keep him as a house cat, which suited him as he was so incredibly nervous and just wanted to hide for the first couple of months.
Eventually with hours of input, he got to the stage where he was comfortable enough to come out of hiding, but was only happy to let one of my dc's near him (with everyone else he would run off if we got too close), and one time when he thought my bf was going to pick him up (he wasn't he was just trying to open a door to let him into the room), he got so freaked out he turned round and attacked my bf, digging his claws into his bare legs and just kind of stayed there too terrified to move!
Fast forward a few weeks later and he was by the kitchen door one day when i opened it to take the rubbish out, and rather than try to go outside, he ran back in and hid.
A few weeks later he was by the door again when i opened it, and before i could do anything he darted outside, and that was that! He would hang out in the garden but could never be caught or persuaded back in the house again, and after a while he didnt even stay in the garden.
The kids were worried so we put signs up asking if anyone had seen him, and we did get some calls, and amazingly he let us bring him in the house overnight, but the next morning as soon as the door opened he was gone again, and hasnt been back for weeks.
We see him occasionally around the area, but he wont be caught or persuaded back. The kids miss him, which is a shame, but I am not sure what else we can do.
We had a cat for 5 years before him, which sadly died after being hit by a car, and never had any problems at all, which should have been more likely, as we found him as a stray or abandoned kitten by the side of the road!