A lot of small businesses don't take cards because historically charges have been high, especially if your average transaction spend is low and people were more happy to pay cash for such items, and even expect to, and you would be considered to be a bit of a twat to try and pay for a transaction of under around a fiver on a card.
A PP above quotes 69 p for debit card transactions and 2.5% for credit card transactions. Years ago DP ran a shop for a while and the charges were similar. Plus you have to pay around £30 a month to rent the machine and further charges for the money to be paid into your business bank account.
Imagine if you run a newsagent, takeaway or bar, where many transactions are for things costing under £1, and nearly all spending being under £10. In a lot of cases, someone paying by debit card would mean that you earn absolutely nothing from that transaction and even if you did, the card provider has taken a lot of your profit. A lot of the 'I use my card for everything and think that I should be able to use it everywhere' people are probably blissfully unaware of the economics of running a small business.
In addition, these types of shops will buy a lot at the cash and carry, so have a way of legitimately spending a lot of their cash, without needing to put it in the bank and pay the bank to do so, so the cost of dealing with cash is further reduced. Of course some people won't declare everything they earn, but many will.
It's now changed a lot and you can buy a terminal for under £100 and pay a percentage on each transaction with no minimum, and I believe that contactless might even be free, plus you've got more people wanting to pay by card, so probably some shops do need to catch up, and change their provider. But the systems need to be reliable, card machines do go down, which could affect takings when no-one has any cash any more.