Why ruin both your holidays, when you can just leave him to it? also, there's nothing to say all 5 of you have to spend every moment together...
make some plans that invilve various people at different times:
~do stuff just you three (You, DH, Baby - go to a park, go for a walk, have a meal together, go to the zoo, go to the beach)
~just you two, leave baby with GPs for a few hours and have lunch together, go for a walk, do something like paddleboarding or archery or whatever.
~Just you and parents, leave baby with DH - spend some nice child free tmime with mum and dad, pottering around, doing an activity together, etc
~Just you, parents and baby - spend some time with baby and GPs letting them spoil baby, or show her off, or tkae to feed the ducks
~all 5 of you (perhaps just do something low key like go to the beach, you can sit in your own worlds if needed, take baby paddling, build sandcastles etc)
~go out on your own, have some "me" time - chill out, go shopping, go for a walk, do something just for you.
why stress out about it all??
Just make sure to let DH know everything he's missed out on whilst sitting in doors playing PS4... 