The thing that really stands out for me in your conversation is that it's all about him - how he feels, his life, his needs, his wants.
Some extracts from your post about the conversation you had are quite revealing.
he's been genuinely very busy
I'm sure he is, we all are in our own ways, but we find time to communicate properly with the person we're seeing.
He said he is very stressed at work
He still finds time to be on Facebook etc though.
He said that it was me and not him who created a situation where our relationship was more casual.
Oh. So this situation is all your fault now, is it, and he's completely blameless is he?
He said that was what I wanted before, and if it's changed I can't just demand he changes to suit me.
I don't see any evidence of you 'demanding' anything. Surely relationships evolve as they go along, so it's natural that you get more serious as time goes on? What he's saying here is he likes it casual and wants it to stay that way.
He said he is confused since I started being more "full on"
Well, all he had to do was ring you and talk it through. But he doesn't want to because he likes it staying casual and he's trying to get you to go back to being less 'full on'.
He also said if I want to talk to him, I should just call him
Oh, so now you have to call him? Even though he's the one that mentioned several times that he would call you on a certain day, then didn't bother. Even though he stated he wanted to hear your voice, but did nothing about it? Looks like you're the one who has to do all the work and make all the effort, Jess. That's because he likes things the way they are - casual - so any additional effort has to come from you, not from him.
He says I made him angry
WTF?! Red flag.
he says I am also cold
Again, WTF?! Red flag.
he said he does find it very hard to express what he feels and I shouldn't get angry because that's who he is.
He's telling you who he is. Listen.
he did make it clear to me that I have kind of ballsed it up
Again, he's making this situation all your fault. He's also paving the way for things not progressing or evolving any further, which will then also be your fault.
He said he was deeply hurt by the constant rejection
What about the rejection you've felt by him promising to call then not, and his vague, non-committal texts?
He says I have to think very carefully about whether or not I really do love him and want him and want a future, because he does not want to invest his emotions any further if I don't.
He's not invested any emotions as far as I can see, unless you count him paying lip service by suddenly saying he's in love with you because you brought up the 'where is this going?' type conversation. Again, he's paving the way for things not progressing any further and he's making his excuses in advance (which will, of course, be your fault).
He's turned this whole situation into something that's your fault and is now saying it's solely down to you to solve the issues and sort it out. Thing is, he's also saying that you can't do that because he can't change how he feels just to suit you. So you're in an impossible situation - damned if you do, damned if you don't. By saying the things he has, he's keeping you dangling just enough so that you don't end it, but isn't willing to contribute very much effort at all into making it work. So where do you go from here?