Ultimately, if kids don't care so much who they go with, there is always the option if finances can't extend, for the SM to go with their child without their dad, since ultimately, it's the holiday alone that matters, not who you are with. Surely it applies to all children and adults?
I personally do think it matters who you go with, not that kids wouldn't love a holiday with grandparents or whoever, but that it's preferable to go with a parent who they are more comfortable with. I don't think most of us are saying that kids would have as good a time with any old body as they would with their dad, but just that their holiday with their mum is not less enjoyable for them because their dad isn't there. Maybe it would be if their parents were together, but they will be used to being with each parent without the other.
I honestly don't think it's reasonable to expect a parent to go away solo with a toddler when they have a partner, and I genuinely can't understand how anyone thinks it's the same? It's a completely different scenario to children going away with one parent.
My DP's ex has recently taken my SCs away alone for the first time, they are 11 and 7 now. She has always taken them with her parents until now because holidays with young kids with no other adults for company or help are no fun at all for the adult. I would have thought most people faced with taking a toddler on holiday solo probably just wouldn't bother, since it's harder work than staying at home.
I think it's totally pointless for OP to be struggling through a 'holiday' with a toddler when her DH is sitting at home alone for the sake of it. And I actually genuinely think it has the potential to create worse feelings for SCs if they arrive at their dad's to be told that their SM has taken their younger sibling away alone for a holiday. That is far more blatantly leaving them out in my eyes.
Also, thinking of my own SCs (and I appreciate not all are the same), if they called my OH from their holiday and found out I had gone on holiday with my child and OH was left behind, their first thought would be of him sitting home alone while everyone else was on holidays having fun. I know 100% this would be my SS's very first thought. They are kind children and would hate the idea of this.