JK, DS dad is pretty much a deadbeat, or has been up till now. He abused you very badly, and still continues to do so.
Why on EARTH are you giving this lowlife creep any space at all in YOUR life.
You need to manage your boy's expectations. He does know why you are not together doesn't he? you haven't sold him a pile of lies to protect that bully? You MUST tell DS the truth, you must say to him that Daddy does love you, but he may not always do what is expected, you are sure he will try, but, but it may not happen. Otherwise DS is going to be disappointed again and again and again. It'll wreck his confidence.
Why on EARTH would YOU want to spend anytime en famille with the Ex? for the love of god, pack this thinking in JK! It's the reason why you are not moving on, twatfeatures still has his tendrils in your life!
If X wants contact with his son, he comes to YOUR house, knocks on the door and you open it, let DS out, kiss him goodbye and say see him later. (or if not safe to do so, at a contact centre).
You don't tell him anything about your life, only talk about DS related matters, nothing else. HE'll use it against you. You seem to have lost sight of what kind of a creature you are dealing with. You don't think he's normal do you? 
You know what kind of stock that man comes from. You know he'll never make father of the week year, even if he wanted to be a better dad, where on earth is he going to get that knowledge? his family? his culture?
Sorry JK, tough talking, but your life is stalled and you are both still expecting X to pull through. He has more chance of walking on the moon.
Give yourself a shake and stop this rot. now.