OP: I hope that you, your son and your dog are all ok. Sending lots of love.
Your teenager sounds like a very considerate and responsible young person and I'm so sorry that his dad couldn't do a simple task to help (move a food item off the floor - it's really not a lot to ask).
Shouting, swearing, threatening, pushing and intimidation (forming a fist) are all examples of abuse. What concerns me as well is that he involved your child in this by calling you both names etc. Horrible, abusive man. Horrible parent.
If he ever does this again OP please call the police. You don't know when he's going to snap and actually hurt one of you.
In the meantime have a chat with a specialist charity like Refuge or Women's aid about what's happened. They will be able to talk to you without any judgement or pressure and take you through your options. If you choose to leave for example, they can help you with a safety plan (but they won't pressure you to make any decisions).
This man is not safe. He coerced you both to apologise to him and you did absolutely nothing wrong. This behaviour is not normal and not okay.
Be strong OP. You, your teenager and your dog all deserve peace and safety.