I want my son to bond with his dad but he doesn't seem to be able to do that and his dad just keeps getting disheartened by this and backing off
The thing is, your son cannot "bond" with a man who is not committed to deepening and strengthening their relationship. By definition, bonding is mutual. By "backing off" your partner is demonstrating to your son that he is not there for him. Children need consistency and unconditional love and acceptance. This man is not providing these for your son; his feelings and actions towards him are half-arsed, and your son understands this. He cannot trust his dad. His behaviour expresses this understanding even though he cannot verbalise it.
In addition your son has had you all to himself for his whole life. You have prioritised him and given him all your time and attention. Now he is having to share his mum, his home and his whole life with his so-called father. His world has been turned upside down, and he is likely to be feeling resentful and insecure.
Given both of these factors it is no wonder "he doesn't seem to be able to" bond with is dad. He is only four! What is your partner's excuse?
It seems that your partner has a resilience problem. He has very little emotional stamina. He has literally abandoned two families, and even when he is physically present, he is unable to put in the hard emotional work needed to sustain close relationships with loved ones. He is immature and selfish.
Unless he is able to self-reflect and improve, he will dump you and your son again. You say that leaving him would be terribly hard - but would it not be easier than having him, inevitably, leave you? Ending the relationship will empower you, it will be your decision, and you can take ownership of your choices rather than being an unwilling victim of his.
And, as long as you communicate effectively with your son, splitting up with his father will show him that his welfare and well-being are more important to you than anything else. I think the "hardest thing" would be continuing to allow your son to suffer and potentially become alienated from you, for the sake of your irrational attachment to a weak and self-serving man.
Wishing you all the best with taking a difficult but necessary step 