Do you live with your family or your partner? Are you financially dependent on your family?
I think it's time to start drawing some boundaries and refusing to get drawn into the middle of this.
It doesn't look good what your partner did and to be honest I'd be not wanting to spend time with someone who had hurt my daughter/ sister like that - but we've had drama within my own family and sometimes for the sake of people we love we do have to bite our lips a bit and get on with it.
I think you need to decide. You can't have what you want - ie your family forgiving and forgetting, your partner coming on holiday with you all like nothing has happened. In some families that might work, but with this particular collection of people it's not going to happen.
So - decide if your partner really is worth all this and you really do want to make a go of it. What is he doing to show that he's sorry and isn't going to make the same mistake again? Is he a decent partner/ father in other ways - and I mean ACTUALLY decent, not just 'well he doesn't hit me and takes a passing interest in his child'. Is he kind, does he do his share, does he actively make your life and your child's life better?
If he is worth it and you're committed to making a go of things with him then you need to say to your family,
'sorry it's just not going to work me bringing baby on holiday without partner there'. You can explain that you can only afford one holiday and want it to be together, or that you don't want to be apart/ take DC away from their dad for that long if you want, or just say no sorry it won't be happening.
Then when they start with the 'well so and so won't go and then we won't go and then noone will go' you say 'don't be so silly, there's X other people going, you'll have a great time and we will see you when you get back.'
When they carry on with the 'no we can't go because you won't do what we say' line then you repeat variations of
- it's up to you what you do about your own holiday
- well that's your decision to make
- I'm not responsible for you choosing to miss out
- well you have to do what you think is right, just like I'm doing
- you could definitely go without me but you're choosing not to, that's down to you
And then change the subject/ leave.
Separately have a conversation about how you appreciate that they were upset by what happened, you know they love and are concerned for you just like you love them. You can understand why they find it difficult to trust your partner, but that you have made the decision to try to make a go of things and you hope they can respect that. You can't keep splitting yourself in two, and if they really can't spend time with your partner then you will understand, but that it may mean you having to skip certain events or spend a little less time with them as you have to do what's right for your own little family too.