Honestly @AnnaTomy what positives are you getting from this "relationship"? This "frugal" man/selfish miser doesn't seem to care for you at all, from your posts.
Why would you have him in your home enjoying all the home comforts whilst he contributes virtually nothing financially or emotionally whilst you work long hours and when he is costing you extra work, mental and physical energy, food, utilities, council tax (because you no longer get the 25% single occupier discount) leaves you with all the chores to do - to which he does contribute - and makes you press him for his part of any holidays which you've initially paid for from your own funds?
It sounds like he's using you as a billet in a long line of billets in the past where he might no longer be welcome.
Sorry OP but surely you are worth more than this?
BTW: Your question provides its own answer.
Am I being unreasonable or is he taking the p?
The answers are NO and YES in that order.
Hope you have the strength to turf him out soon.
Also could you bear to have a regular lodger (post lockdown) to bring in some extra money to top up your retirement fund?
Best wishes for a better future. 🌷