A conversation with his partner is not going to take a lot of time out of his precious, busy life.
It's not about the physical time needed but the mental capacity required to think, fully understand yourself and your position so that you know what you can offer and what you can't. Otherwise the conversation will progress nothing, achieve nothing and is likely to end up frustrating them both without a resolution.
So what he is saying/asking for isn't wrong but, equally, what the OP is asking for isnt wrong either.
He's at the end of finishing a PhD as well as working so he is under a lot of pressure but it doesn't make me feel like a priority.
The reality is that, currently, you are not his top priority.
People on MN like to say, "If someone loves you, they want to see you/spend time with you," and, "If he wanted to, he would," but in the real world it's not always as simple as that.
I sometimes think well, yes, if you work set/regular hours in a job you can just leave behind when you walk out of it at the end of the day it's probably a more reasonable expectation but not all roles are like that.
When I'm in a really busy period at work, I simply don't have the time, capacity or headspace to consider anything else. I've always made that clear to men I've dated (and friends/family tbh) and I know it has been an issue for some. Others have just used that time to do their own thing and catch up on their own lives. And I certainly wouldn't have the capacity to travel 3 hours each way to spend a weekend with someone.
That doesn't mean it has to work for you, though, OP. And it's fine to end it if it doesn't suit you.