Lets break this down:
Honestly the age thing doesn't bother me. He doesn't even look like a 36yr old. Irrelevant
Anyway its only been 3 weeks and we've seen each other quite a lot.
But its always at his place although he usually swings by a shop to get me something to eat.
Wow... Big of him.... he's not even bothering to wine and dine you, just bringing offerings.
I feel like its moving fast but he disagrees.
you feel it's too fast - that is ENOUGH. You are entitled for this to move at your pace. regardless of whether you are 19/29/39 or 49
He DISAGREES? Bin him for that alone!!!!
Its physical although we haven't had sex. He's already asked me to sleep over and got mad when I refused.
Red Flag love, bin him now.
He's really nice to me when we're together but when away from each (through texts) he's not as nice. He seems frustrated/angry etc.
HE IS NOT NICE! NOT AT ALL NICE!
And isn't happy about us not having sex, one time he looked like he was gonna cry. This is the first guy I've ever been with like this (intimacy).
HE HAS NO RIGHT TO DO THIS TO YOU LOVE, NO RIGHT AT ALL. Whether or not you are a virgin isn't the main thing here, (although extremely significant in many ways) he is refusing to listen to you and respect your decisions.
End this today. Get rid of him and never ever look back.