My BF has a 5yo, he has him every week Thurs to Sunday. I have a 6yo, I have him full time.
To begin with we saw each other on Sundays, some fridays. Now, 3m on, we tend to meet up all together on Saturdays, he comes over to mine on Sundays, with the odd evening here and there. It IS hard for us to get some serious alone time, but we're working on it. It helps as our children are similar ages, get on really well and love playing together, otherwise it's make things tough to arrange, and expensive too with all the babysitting, we could go for weeks without seeing one another, and that is not what either of us wants.
Your situation is more difficult, fwiw, I do think he's being a little unreasonable about not planning time with you, at least on the weekends that you are child-free. He does need to make more of an effort. Have you spoken to him about this? CAN you pin him down to plan some things a couple of weeks in advance?
He has to make time for an adult in his life, his kids need to see this and will understand this.