Oh dear, your DF isn't coming across as the lovely person you first mentioned him to be @amiungrateful. He sounds like he is being "older person selfish", which I feel ok about saying because I am an older person too, and I know the "draw of being more selfish" now that my next big birthday will be my 70th - if I am blessed enough to live that long. I hope I am not being too sexist when I say that, especially for my generation and older, many Dads and Granddads who have lived in a more traditional way, just don't understand the needs of Mums and their young children.
I do think you are being rather unreasonable and unfair to yourself, your DToddler, and your DH, by not having a (probably) long and frank discussion with your DF, about the parental pressures he is still putting on you as an adult and parent, in your own right! I can't help but wondering if that is because he has some sort of financial hold over you? I don't mean ro be rude, but you are a stay at home Mum, who can still afford to have a housekeeper, you can afford what help you want on a holiday, and you can afford your plane ticket back home if you wanted to go home early. All of which suggests to me that you maybe followed in your DF footsteps, and were able to make a lot of money before you gave up work to become a SAHM, or maybe you inherited a lot of money, or won the lottery, or perhaps your DH has a very well-paid career, or your DF highly finances you? If it is the latter, I could understand (if not personally agree with) why you feel you have to keep him "sweet" by never standing up for yourself or your immediate family.
However, if my last suggestion that you may (all) depend financially on your DF, is anywhere near the truth, then please take this old woman's view on board, which is that unless you are going to be left very poor and struggling without your DF's money to enhance your life, then the freedom to be yourselves, and make your own decisions, is worth far more than continuing to live a fairly luxurious life - in comparison to most of us anyway.
If you don't have to take your any of your DF's wealth into account, then please just explain to him - in great detail - how and why you feel the way you do, just like you have explained it to us here. You could also ask him if he would be happy to leave his girlfriend for two or three weeks when he goes on holiday with you, and if he wouldn't be happy to do so, why does he think you and your DH would be happy for you to do so, and it would also mean your DH not seeing his DT for two or three weeks too. Please just tell him that two weeks is your absolute limit, and only then if your DH can join you for at least one of those weeks, and again, only then if the week you are without your DH, your DF babysits your DT for at least two evenings, while you go and have a nice dinner with his girlfriend, and during the day he plays on the sand with his DGC for 20 minutes to half an hour, so that you can have a proper swim in the sea yourself.
On top of all that, please consider slackening off your brilliant DM mantel while you are on holiday, and just be willing to be a reasonably good one instead - if you can keep your DT happy in a restaurant if she can watch your mobile phone's screen (not loudly of course) then please do that. You also don't need to stick rigidly to the same routines that you have at home, most little ones can adapt, and then adapt back again once at home, quite easily. I hope you end up having a lovely holiday OP, and that you can come to a much better understanding with your DF 💐