About 4 years ago FB sent me an Email, with a list of people I 'knew' who are on FB, in an attempt to get me to contact them. OK you might say, what's wrong with that ? - except I am not and never have been a member of FB. Turns out that one of my friends, who was on FB, ticked a box allowing FB to trawl her email account for any email addresses which it could then match to other people on FB that she knew. She did not realise that it was also snooping for Email addresses like mine, for people who were not signed up to FB so that they could try and get them to join. So it basically harvested her Email account to get info on people not signed up to FB.
The people it said I 'knew' on FB included someone who had harassed me with emails about 5 years earlier, and I had deleted every email received from them since. FB had just decided as we had had previous Email contact it would encourage me to join FB so that I could get back in touch with my 'friend'.
I sent several request to FB to remove my details from their records as I had not agreed that they should have them, I did not ever receive a reply back.
So, YANBU OP, if is not just a question of they know about you if you have signed up, they appear to be collecting data on people not signed up either, and that is not acceptable IMO.