You can understand that your partner’s two older children are jealous of your two with him and will want time alone with him to make up for the six lost years.
So it’s difficult to know what your dilemma actually is, -with regards to their jealousy.
The clear answer, which you must have already have come to, is for him to have enough time with them now, to satisfy their needs so that they are no longer jealous. And for this to happen, there have to be adjustments to all your life styles so that your partner is free to make it up to them. He needs your help for that and it has to be talked through, not just assumed that will give it.
I know you aren’t married to him but still you are in effect, hopefully a potential step parent to his older dc. As his partner, it’s important to work with him in any way you can, stay as a strong team together, so that you don’t add any extra obstacles to them.
Having children with a man who already had two children was never going to be a life without extra problems to overcome.
I hope you can manage to find a way forward, but it will take a lot of compromise and not be easy especially if the ex wife is difficult and controlling. Good luck!