OK, kill with kindness, say, No, because you have plans. Say that with more notice of the mega-sleepover you could have planned your Sunday accordingly, but it's too late to change.
If Ex gives more notice, of course you can consider it. Breezily reply, next time, we'll sort it.
As you say, DS has regular contact with his cousins so not really a once in a blue moon, have to move heaven and earth to make it happen kind of scenario.
Abusive men often have this nicey nicey excuse/reason to make you do what they want you to do. On the face of it, it's designed to make you look like the bad guy, so you give in.
Be firm. Your DS has to understand that plans are plans and if there had been more notice, and there were no plans in place, then it could have happened. Push it back on to his dad, by saying that next time daddy needs to be more organised and tell you sooner.
Hugs, I know you feel like shit about this, but given your past, your instincts are probably right.