Why should I have pain I don't want inflicted on me too?
This works both ways, of course.
Some would say hardship has been inflicted on many people due to our country having been in the EU. It may be financial (e.g. lowering of wages, or funds from British taxpayers going to what the EU prioritises, not what the UK decides). But it can also be how people feel about their sovereignty and democracy being transferred away, or the way the EU has stunted some parts of our country's enterprise. E.g. they prioritise large corporations who can afford to lobby the EU constantly and get the rules made in their own interests, at the expense of smaller, and perhaps more innovative, companies (Dyson being one example).
Creative fields have been influenced by the way EU funds are distributed, as of course only EU-approved projects will receive funds which could have been distributed to the UK's different choice of artistic endeavours. So arguably that is a harmful reduction in our cultural wellbeing.
It's a valid argument to believe you cannot put a monetary price on democracy, sovereignty, independence etc. People have given more than money in the past to stand up for those.
"EU citizenship" was forced on every British person in the 1990s whether they wanted it or not, and you can't even give it up without simultaneously having to revoke your UK citizenship. The price in return for this invented proto-nationality (ready for when Britain would be just a region of Europa) was the ratchet effect of one-way flow of various aspects of control of our own country, away to Brussels. No surprise that working on the Maastricht Treaty had been Jean-Claude Juncker.
The EU has prevented the UK from developing more trade worldwide, and held us back with outdated approaches.
The EU resulted from the CIA's ambition of the 1950s (as shown in declassified documents, see here) rather than being something Europe would naturally have developed.
The plan was always to create a European superstate by stealth, and this secrecy is undemocratic. At first, the American Committee on United Europe sent funds to the European Movement and the European Youth Campaign. The transfer of money via the ACUE to European federalists was to continue, to groups such as the European Coal and Steel Community. A memo to the vice-president of the European Economic Community recommends suppressing debate until the point at which "adoption of such proposals would become virtually inescapable". The sense of loss of the UK's self-determination as it has been absorbed into this agenda is also something which some people consider very detrimental.
The perception of those you disagree with causing difficulties you didn't ask for works both ways, and is definitely not just about money.