Your polling card identifies you to the officials inside the polling station - it is not essential - you can vote without it; I have - all you have to do is give your name and address. It provides you with information about the date and where to vote and is entirely apolitical. Nowadays without it you might have to produce photo id I image otherwise you could be anyone.
The people outside who ask for your card are "tellers" an whether you give it to them is optional. They represent all the parties. They are there because they are members of the party faithful who will have been canvassing in the weeks before the election. Canvass cards are taken from the electoral register and marked by name, road and house number. People are canvassed to determine their support. If you promise to vote Labour, for example, a note will be made of this on the canvass card (the information is kept by your local party). You therefore become a known supporter.
The tellers are collecting information in relation to their know supporters. The cards or lists (all the reps outside share the info) are taken back to the action room for your ward. They are compared to a list which notes all that parties supporters. If it is noted that a known supporter has voted then manpower will not be used during the day to deliver further leaflets to you or to knock on your door to encourage you to vote.
That's why I always vote early and always make the info available. It means my party can use it's manpower knocking up those who haven't voted and knows not to waste time on me
At every election the parties canvass and keep records so that on polling day they have an idea of who their supporters are: who supports other parties (party workers don't waste time on the doorstep talking to known supporters of other parties), and the "don't knows". They will keep revisiting the don't knows and the unknowns during the campaign to try to convert them.
The tiny number on the polling slip is different from that on your polling card. Once the votes are counted there is no exercise to match the number on the slip with the number on the list. It could be done but I can't think of any time in UK history when it has. It is to make sure that no fraud takes place and protects democracy and freedom rather than harms it in any way.
When the man said he'd knock on your door - all he meant was that you might be knocked up to go out an vote all day - waste of your time and of his.