Is he Canadian?
If he's being made redundant he'll have no need to return to the UK and may try to remain there with your child. And if he has a chunk of redundancy money, it's more likely.
Many years ago I worked for the then immigration service.
One of the things we did was operate a watch list of children likely to be taken out of the country. It was a problem. Once a child is out of the UK, especially if the taking parent is a national of the other country, things can get very difficult.
But regardless of his nationality, say no to this trip.
Three weeks is a long time for a toddler to be away from his mum. It's a controlling tactic; to cause as much distress to you as possible while on the surface seeming to be doing something 'nice', a holiday with daddy.
Do everything in writing, not in person. Easier to bully and intimidate in person.
Yes, accept it will mean he will refuse permission for you to take your child abroad on any holiday in retaliation, and may do this until he no longer has control. It's a small price to pay and you'll have some lovely holidays in the UK!
The first thing you need to do is get him out of your head. Therapy to help you move on from an abusive relationship. A network of family and friends. He's still working away in there, prodding the bruises, making you think he can control you, your home, your childcare, and you need to heal yourself to be strong for your DC.
You can say NO.