Scruffy i know exactly what you mean. Its really hard if you're not the type of person that just shouts into someone face, trying to be polite and that, but with some men it just doesnt work.
There is a man in my neighbourhood, a cab driver, that stops to have a chat everytime he sees me, first time he did, he said i resembled some one he knew, we a bit a bit of polite conversation that lasted aboiut 1min and went our separate ways, or so i thought, i seem to bump into a lots in our local tescos, and he starts smiling, sickly smile and tries to start having a chat, it got to the point that i was dreading going to tescos. It wasnt just in tescos that i saw him, he seemed to be everywhere. This went on for about 2yrs.
one day i was crossing the road to my house when a car horned repetitively as as i looked up i saw a hand waving and quickly waved back, assuming it was a someone i knew, only for him to stick his head out of the car and it was this man! i was furious.
shortly afterwards i went to the station, and who would be parked outside ? this idiot, smiling and beckoning me over, i went over and said to him" DO I KNOW YOU ?" to which he answered "remmeber when i bumped into etc, so now we are friends!", No we are not friends and i dont know you i retorted.
I must say i was shaking after this confrontation, but i havent "bumped" into him since. In short tell him to GET LOST!