OP, if for 20 weeks you only get tiny amounts of time with eachother, he doesnt love you. Because from what you say, you dont spend enough time togther to be in love. You dont really know eachother.
He certainly doesnt know you are jealous and resentful of the time he spends with his kids. So he doesnt know you.
You have carried on seeing him for 5 months knowing the routine. His assumption is that you are happy with this.
So talk to him. Be honest with how you feel. 2 things will happen. He will tell you he wont change anything. Or he will compromise. Then you make a decision.
If there are to be changes, I would say it shouldnt be less time with them. But different days. So every other weekend and he should change it before you meet them. If they meet you, then everything starts changing, they could end up resentful of you.
But let's say its every other weekend and days during the week. You really need to think about wether that will work for you. If you move in, half your time will be spent with these kids. Your home will be their home too. When/if you have kids of your own, the home you share with their father and your child will be their home too. You will potentially have 3/4 kids there half the time (depending on how many you have). He will still need time on his own with his kids. Will you be happy with him going off for some quality time with them? Will you be happy with your routine being based around them too. Will you be happy having them last minute because the mother has asked? Not saying she will take the piss, but good co parenting includes helping eachother out occassionally.
If you really believe you are in love and thinking of this longterm, plus are aware you can be a bit selfish and are jealous of the kids now, you need to be thinking about this now.