If you didn't have kids and/or you had a rock solid relationship, I'd be more tempted to say give it a go.
But the things you've said about your relationship do not say good things about it or your partner. You've described some really poor behaviour on his part, and I'd be taking this time to think about whether you want to be with him at all. What do you get out of being in a relationship with him? What does he contribute? How does he help your son grow and develop, and what behaviours does he model when he is around?
And, even if you do stay with him, given that you have a child (and, especially, an already shaky relationship), I definitely don't think you should move to Australia with him. Moving countries is stressful. Finding a house, new jobs, a school, childcare, new friends, sorting out doctors and dentists and a new healthcare system, all the immigration paperwork. It's not easy even if you both really want to be there. It puts strain on solid relationships. It can break them, especially if one partner settles and is happy, and the other is miserable and feels isolated.
And if you hate it there and want to move back to the UK but your partner objects to you taking his child out of the country away from him, which he might even if he doesn't want custody, the courts will probably decide it's in your child's best interests not to be uprooted from their life in the country they call home, whether or not their parents have split up. So you would either have to stay, miserable, or leave your child when you move back to the UK - which you probably wouldn't want to do.
Don't move to Australia. Tell him if he really wants to do this, he should go without you and your son, try it out for a year, see how he does, whether it's what he expected and hoped for. It'll give you both time to think about your relationship, if nothing else, and the practicalities of the whole situation might make him rethink.