OK, so YOU are the parent, if you see someone deliberately hurting your child, cos he can, what are you going to do about it?
Your DS clearly is going to struggle standing up to this bully, so YOU have to.
WHY NOT tell him that under no circumstances is he to treat your DS like this, that it is cruel, hurtful and harmful abuse and you will NOT allow him to do this to your son.
TBH, not holding endless grudges against a truly heinous 'father' like this will actually cause your DS more strife, hurt and damage over the course of his life time. His esteem will suffer, he will be hurt, damaged and it will have adverse affects on his character.
Until you can teach him to stand up for himself, repel this negative and toxic parenting, then tbh, it's ill-advised to allow contact.
If you don't tell this clown how it is, then you are somehow letting it slide. OK so he might not ever change his behaviour, but he won't ever do so if you don't call the fucker out on every single shitty thing he does to your darling son.
Put your son above everyone else and fight like a bear for him. It'll show him that his father has no right to behave, and it'll show him that there are some things WORTH kicking off about, even if it won't make a twat like this change.
To say nothing, on some level, normalises it. Your DS deserves wayyyy better than this prick.