No, especially if the pricing reflects the lower running costs that come with being cash only or the service is really good (i.e. efforts are fociusse don making the service the best it can be not on being slick).
There are a few cash only small business near my flat (very centrally placed in a reasonable size city). Small independent fish shop run by an older guy, little cafe specialising in home baking done on site, a coffee place. All very good, focussed on their core service and usually no computerised tills etc to pay for either.
They are often noticeably cheaper than comparable local/national businesses that take cards. They are also very simply run, one trick ponies who do their one trick well enough that they keep pace/stay ahead of "slicker" rivals. Usually sole traders with no or only a few employees.
There is also a fish delivery business who will deliver any order over £5 if you phone the day before or leave a answerphone message overnight and a laundry that drops off and picks up washing/ironing/dry cleaning- both cash only. Fish delivery place has a huge range and is great quality. The laundry place is a lot cheaper than it's national chain rival, and the finish is better (i.e. they bother to spot clean stains etc).
There is also a very, very good corner shop, a good newsagent who take cash only- and they both have fee free cash points within the store, which is a good solution I think. The very, very good corner shop will extend credit to regular customers who forget their purse or are running a bit short- I've often overheard them say "oh give the money next time you are in" over a few quid. They restore my faith in humanity sometimes to be honest.