It'd be a resolute "no" from me, too.
My ex was an active/involved parent to our son as he grew up (he's now an adult, so sorts any contact with his father out himself, and has done since he was 15 and I took a huge step back because of the stress my former "in-laws" were creating...) - but he has never spent a night in our son's company from birth. Son is now 19. When he was 6, his aunt got married at the other end of the country - we live in the SE, the wedding was in Yorkshire sort of a distance. Obviously, she invited my son and yes; it would have been nice if he'd been there... but there was no way that I felt comfortable about allowing it.
Firstly, there was no inclination on ex's part to even start the process of overnights with our son. He had suffered a traumatic brain injury at the hands of my ex-MIL a few months prior to the "oh, yeah, [SIL] is getting married and I want to take him hours away from you, for 4 days... it's going to be great; we're all planning on getting absolutely hammered!" conversation, and consequently, he was having night terrors and was extremely clingy to me/his nightly rituals. Ex had no clue how to deal with either of these things... because he didn't ever endure them (and I don't use that word lightly). Our son slept in my bed for months, as a result of the TBI. Secondly, it was dropped on me with short notice - if I remember right, maybe a fortnight before the wedding, like DS was an afterthought (ex had a toddler with the OW at the time and, possibly, a new baby, so I could be gracious and go "sleep deprivation, ex simply forgot..."). It felt like DS was required to look cute in the photos. Had DS' older (full) sister been invited (who was 13 or 14 at the time) I would have had more confidence in saying "yes" - familiar face for him, she knew his routine, would have been sober and therefore capable of keeping him out of mischief whilst their father, uncle and grandparents got "absolutely hammered". But she wasn't. At all. Don't actually know why, but suspect it was because she was furious with ex about the way he told me he'd been having an affair (on the day the then-toddler was born)... and she was at that awkward "Ugly Duckling" stage and wouldn't have fitted ex-SIL's criteria for the photos... Secondly, ex refused to be on either birth certificate and paid £5 a week maintenance towards their upkeep.
Surprisingly, perhaps, given the getting blotto at weddings/assaulting my small child/photo prop situations, his family are middle-class (but think they're upper-class 😂) and I couldn't trust them with my child's safety. So I said "fuck off" in a very polite fashion, and gritted my teeth through their whining for the next nine years. Not once did they ever suggest what they could actually do to permit me saying "yes" to DS going on holiday with them. They did, however, write in his 18th birthday card (which arrived 2 months late...) that "now" he could get a passport and "go on holiday with" them. DS has a passport, they knew he had a passport because I've taken both of my children abroad - as has my mother, without me... and it's never been mentioned again. Despite their having been off skiing and to visit relatives in sunnier climes, several times. And yes; DS would go with them, if they paid. They expect me to do that, however. So in hindsight, I am still confident that I made the right call.
@MissNP196 - you're absolutely doing the right thing. Just keep a paper trail to prove that he's all mouth and no trousers, as they say. And know that because you are putting your baby's best interests at the core of this situation, and he blatantly isn't, and you're not telling him an outright "no", just "not right now, lets build up to it..." you are actually showing him what being a real parent is all about. Your DC will notice who is the one parenting them... and who isn't. Mine definitely did.