You said you weren't up for it, he should have backed off straight away. Not necessarily because he got the wrong idea... but because consent can be withdrawn at any time. So, no, you weren't in the wrong to say stop, and you weren't in the wrong to be upset that he didn't immediately back off and apologise.
Going forward though, OP, I do think you have been a bit naive. I wouldn't go back to the home of any man who I didn't know well - as a broad rule, men are bigger and stronger than us. Men on apps are men who could be anyone - you have no idea they are who they say they are.
Talking about being 'too warm', talking about your underwear, suggesting to or agreeing to going back to a man's home... those are all suggestive things, especially in a dating context where the man is looking for signals he can jump from trying to impress you to sleeping with you.
Maybe you would joke about your knickers normally, but you have to be careful when you are getting to know a strange man. He won't know that you don't mean anything by it, so you have to sometimes hold back certain things.
A walk and a coffee is supposed to be a quick, cheap 'screening' type date. By suggesting you went to his place, you were changing the nature of the date.
Knowingly or unknowingly, I think you gave off a lot of signals. That's something to reflect on, so you don't misrepresent your intentions next time you go on a first date with someone.
Based on my experience, if you aren't willing to have sex by the third date, the apps men will turn nasty. It's like there's some secret rule that they are entitled to sex by the third date at the absolute latest. 🤷♀️ Who knows... is there an 'Incel's Handbook' going around?
I've had some lovely first dates, even lovely second dates... but every third date has ended in some downright nasty behaviour when I've refused to go home with them. (And no, I haven't made them pay for everything - I'm very careful to make sure that it doesn't feel like I 'owe' them anything.)
I don't think it even matters which app you use. I think there is a significant number of men on all the apps who live by this rule.
It's not victim blaming to point out that a lot of apps mens have expectations, and to provide advice for how to minimise the odds of being in an unsafe position. It's only victim blaming if you blame the woman for what actually happened, and I don't. OP, you were naive, you did some risky things, but ultimately, as I said at the start, both parties are responsibility for ensuring there is enthusiastic consent. He should have stopped sooner based on your body language (trying to push him off) and he sure as hell should have stopped sooner based on your words.