If you forget or lose your polling card, you don't actually need it on the day - your name and address will be on the list at the polling station. It might make the process a bit faster if you take it though.
I've been a teller on the door at the polling station - all we were supposed to do was politely ask for voter numbers so that we could check which party members/declared sympathisers would need to be dragged out reminded in the evening. We weren't really interested in anyone else, so if you haven't made any promises on the doorstep, there's no point giving your details.
Anybody who wants more information than that is out of order.