There are so many things to unpick here.
Firstly, has he ever said anything like this before? I strongly suspect from your description that - as PP have said - that it was just a ploy to stop you putting across your opinion for fairer finances.
Secondly, it doesn't even make sense. Because other people don't like you, you're not allowed an opinion on your finances?! Other people's opinion of you makes absolutely no difference to your financial arrangements.
Thirdly, I very much doubt that it's true. You might not have a huge circle of friends but not everyone does. I bet when you're relaxed and with the right company, you're lovely.
Fourthly, who cares what other people think? As long as you're not deliberately spiteful to people, then it doesn't really matter if you're not universally popular. Some people instantly attract others while some are a bit of a slow burn. I've known people who I thought were cold and standoffish but over time, as I've got to know them, it turns out that they're lovely, warm, kind souls. Providing you have a network/circle of family/friends that you're happy with - no matter how small it is, then the opinion of other people is irrelevant.
Lastly, who the actual fuck does he think he is talking to you like that?! No one deserves to be spoken to like that. No one. I hope he hasn't knocked your self-esteem so low that you feel powerless.
I'm AuDHD and it takes me much longer to make proper friends. But that's OK. Don't feel bad about who you are.
If it's out of character for him, I would expect profuse apologies and an explanation of why he said what he did. If it's not out of character, I'd be out of there.
Sending hugs.