I work in this industry. If it was priced on the meter then yes he can charge for waiting for you to get the money. If he helps you load/upload shopping he can also charge for that, say you have 4 bags and the meter is set at 10p a bag there's an extra 40p, The meters are set according to local authorities.
Can your house be blacklisted? To an extent yes, we have customers that are banned for not paying etc but it would be unusual for the sake of £1. When these customers ring it pops up on their number that they are banned.
Charging for extra services that you ask for like getting bags out is one thing, but I don't see how he can charge for waiting to get the money when OP had that money right there, in what she had been advised was a valid form of payment. If I called out a plumber who charged by the hour and he turned up having forgotten his tool bag, or if he'd broken an essential tool whilst doing the job before mine, I wouldn't accept him charging me for the time it took him to go back and fetch the bag or to go to buy a replacement.
Incidentally, does that mean that disabled people who might need more time and help getting in and out and with their bags routinely have to pay more for a taxi than an able-bodied person (who might have been easily able to get the bus, anyway), by order of the local authority? Not saying the driver isn't entitled to do so, but I'd feel terrible if I ran a business and charged less able folk more, essentially because they're disabled or elderly etc.
When you say it would be unusual to blacklist a customer for the sake of £1, surely there's a big difference in a customer refusing to pay it and the driver being unable to take the agreed payment as offered. If it were me, I'd 'blacklist' the driver and, next time I called the company, ask for any car except number 31, who was abusive to me last time.
Of course, we can't know, but I too am a bit suspicious about his card machine being 'broken'. Sometimes getting a signal isn't straightforward and you might need to turn them off and on again to reset them if they glitch, but do they really 'break' as often as people say they do?
As has been said, cash doesn't incur a bank charge and, if desired, could be taken 'off the books'. Nothing to stop them putting up a little sign in the car saying "Cash preferred if possible, please" but accepting all agreed forms of payment graciously.
It's pure conjecture, but I wonder if he was hoping that, when he told you it had to be cash, you would ask to stop off at a cash point (all the while on the clock, naturally), withdraw £20 and then, on being told it was approaching £15 anyway and feeling bad for keeping him waiting (even though you were paying him to wait), would just give him the £20 note and tell him to keep the change. Just maybe, although it was a £12 fare, he had mentally rounded that up to £20 and then, when he 'lost' £7, that's what made him angry, albeit knowing that he never had a right to that money in the first place.