No it does matter.
Because the very first thing considered by the Police or CPS in the offence of harassment is:
(1)Every individual has a right to be free from harassment and, accordingly, a person must not pursue a course of conduct which amounts to harassment of another and—
(a)is intended to amount to harassment of that person; or
(b)occurs in circumstances where it would appear to a reasonable person that it would amount to harassment of that person.
And neither of these things apply here.
Somebody knocking on your door twice is nor harassment. Regardless of the 'feelings' of anyone. We have no evidence that the knockers are intending to harass the OP and as you see from this thread, most reasonable people do not think it's harassment to have someone who thinks they have some important information for you or important questions knocking on the door twice when the person refuses to talk to them as 'harassment'.
And that's without the other elements that make up harassment in law. This is just the first hurdle.
It's SO important that people understand the law and don't just throw around legal terms without understanding them.
And equally as important that people don't waste Police time on things that aren't criminal offences because of how they 'feel'
We have an under-resourced, under-funded Police force as it is.