'I take DS round there to see him and he is fantastic with him but if I don't take him round he doesn't ask about him or anything. I don't understand it and it really breaks my heart that he didn't even mention him today'
This relationship is all about you and your absent father. I don't mean this in a hurtful way, but you have work to do on yourself, Fairy.
He gave you a few days of being 'nice' and failed to move you. Then he turned to harassing you at night and angry accusations.
His accusation that you are having an affair is not based on any facts or anything anyone has told him. He is saying that because you didn't answer his texts and calls while you were doing a perfectly reasonable thing all night -- you were sleeping, FGS.
To use an affair as an accusation shows that he thinks of you as property. It also shows that there is nothing reasonable about him, nothing rational either. People sleep at night and don't stay awake in case a violent drunk wants to chat. He thinks you should have.
His abusive texts and his anger with you show that he has absolutely no grip whatsoever on reality or what a relationship should be about. He is not going to apologise. If he manages to say the words it will only be to gauge your response and see if that would work to achieve his ends where bullying might not.
He thinks you should have responded because he is a bottomless pit of need that you can never fill, and because you can never fill that pit he lashes out at you in contempt and anger. The contempt is there because you try. The anger is because he knows he can't be helped. There is nothing, good or bad, that you can do to this man or for him that would make any difference to his response to you.
Be very careful here. He can switch it on and off again with the DS too, not just with you. He can play doting father but he is just as capable of threatening to hurt your son to gain leverage or put pressure on you.
In many ways, you do not exist as a real person for him. You are a screen onto which he projects everything screwed up that there is inside him. Same goes for the DS.