Boundaries OP, boundaries.
The important thing with boundaries is that your boundaries don’t require anyone else to do/not do something. It’s what you are willing to accept. What behaviour are you willing to put up with? What are you prepared to accept on behalf of your son? What does he have to experience?
Your FIL sounds like a toxic bully and MIL sounds like she’s been conditioned to tiptoe around him to keep the peace.
You will only be guilt-tripped if you let yourselves be guilt-tripped. It’s all a choice. FIL is choosing to be a total a-hole even though you’ve spoken about it and made it clear his behaviour is unwelcome, MIL is choosing to ignore his awful behaviour and also choosing to side with him, you and DH are choosing to put up with his behaviour.
The examples you’ve given are pretty bad. The fact that your 3 year old is angry/sad at what FIL is saying is very illuminating. Children that age have “pure” reactions to emotions ie they haven’t had years of experiencing toxic crap so they haven’t learnt to keep people happy, change their behaviour to avoid confrontation, kowtow to the abuser, accept toxic treatment. How your son reacts is how anyone should react if they weren’t constrained by worrying about kissing off WW3 or having the PIL go NC with you. The threat of NC is an abusive power play btw. “Do as I want or else I will punish you”.
If he wasn’t your FIL, would you put up with his behaviour? What he’s doing is controlling emotional abuse. It is damaging. Have a chat with your DH and decide what you are willing to accept. Your FIL isn’t going to listen to you so you need to find your line and stick to it. If you don’t see him that often, he won’t be missed. If MIL is going to stop speaking to you because of it then she is choosing him over the rest of her family and will have to live with that. You either continue to get irate every time you see him and watch your son get upset or you put in some boundaries and refuse to accept his behaviour.
Good luck!