Is this a two week family hols with your own family unit or just a mini break. Is that what you mean by mini holiday? A mini break is less of a transgression.. but a two week holiday its a major committment and should not be going ahead unless you were both in (real) agreement.
I'm so sorry to say this OP but you need a complete rethink
It's very clear that you DO mind them coming... but instead of saying that and making your feelings clear you are talking about just sucking it up and going along with this. I think that you have been used to helping and facilitating others but it is OK to put your own feelings and priorities First. Its not rude or uncaring and you have to be firm about calmly and confidently asserting your right to do so, or you will be walked all over. I've been in your position and it was very difficult. I would never let it happen to me again without joint agreement.
You compromised by suggesting a solo family holiday later on, (reasonable) d and he fobs you off with "it might not be possible (feebe that he can't even give a flat no, but has to wrap it up in a maybe - whilst the inlaws on hols still remains on the table)
In your position, I would be reading him the riot act for not consulting others. Unless he had a really good reason for doing so.
Then I read that YOU are
" the one organizing the holiday, I am the one paying for it. I value our family time a lot, because we rarely get any. " and that he can't pay for it has he has too many debts.
Which I feel makes his actions even more of a cheek.
First I would TELL him, not suggest... that he cannot just invite people on the holiday you organise and pay for without consulting you.
You are absolutely correct that family time is precious and the only times. Family holiday times, away from the normal run of things are also precious.
If he has debts and you have to fund the family holidays (as well as organise them) he doesn't get to go around inviting other people. His debts puts the burden of family finances more firmly on you. You are within your rights to say "We can only afford to take ourselves." but also. I only want to take ourselves.
Yes you can uninvite them because it was an ill considered invite in the first place. If you are organising it.. you can say that you've looked into it and you cannot find or afford bigger accommodation.
If he won't say it to them without fudging it or blaming you, you can tell them yourself on your own terms.
It is not being mean to them, it is not unreasonable for you to want rare time away as just your own family unit.
If he did this because he wants to spend more time with them, you don't have to go away with them for two weeks to achieve that, invite them over for a short visit instead or go to theirs for a short visit and take them out to dinner.
If no one wants to meet you halfway, you also have the nuclear option of just not organising it at all