The problem is that some vigilantes are very stupid indeed, and they don't trouble themselves with obvious facts. To them, it will be 'the paedo house' and it simply won't compute that the house itself is not a child abuser and the new people who have moved in are completely unconnected with the previous occupants.
Look at that case from a number of years back where the single-brain-celled self-styled 'paedo-hunters' heard that somebody local was a paediatrician and so scrawled threatening slogans across their home and tried to drive them out of town.