It is the distance that makes it more difficult.
When my ds's get invites related to time they will be with their dad I pass the invite over to him - totally his responsibility to deal with the invite, from deciding (hopefully with dc) whether they will attend, responding, sorting out a card/present and getting them there and back.
Sometimes they can go, sometimes (if there are already other arrangements) they can't - exactly the same as invites they get on when they are with me.
Occasionally invites cross over a normal handover time - in which case I discuss them with their dad and we agree on the arrangements.
I wouldn't dream of accepting an invite for a time they with their dad, nor would I make any promises to ds's at to whether they could go to it.
But, we only live 20 mins apart - so it is easy for them to attend from either house.
Personally, from where you are at the moment, I would take dss to the party - but make sure that his dad makes it clear to his mum that in future any invites that occur in the time he is with his dad are passed on for his dad to deal with - that his dad isn't prepared to have his mum dictate what happens in his time with his son.
And then enforce that in the future if his mum choses to make the decision herself, by not taking him.
I would then make sure you do as my ex does - take over full responsibility for all arrangements concerning the party.