Please try not to be upset OP, and don’t doubt yourself.
I feel for you, working 7 days a week to give priority to your clients, and I imagine at times you have worn yourself into the ground doing it.
And then someone upsets you like this.
Others will be along with good advice, but if it helps you, in my experience it’s often wealthy people, or those that have had no commercial experience in their lifetimes who are quickest to criticise - when they have no comprehension of business, overheads, tax, insurance, NI, running a vehicle, and all of that before you have fed and accommodated yourself and then turned up for the job.
Are there any other self employed people in a similar field that you could network with - always useful to compare and exchange views, rates and thoughts - and will give you assurance that you are on the right track (when someone like this tries to unbalance you). Could your hourly rate be a bit higher?
One client “told me” that I would need to re-think and reduce my (already modest) rates! - I had rehearsed for such conversations and replied unemotionally that providing the service to the lady at any lower rate would simply be unviable after tax, insurance, NI and overheads. I was valued more as a result after that. (looking after animals rather than people though)