Why would you fight to stay with this man?
Just looking at your list of criteria for getting back together.... random drugs tests / not seeing the woman he had an affair with (and still lies about despite you knowing the truth) / looking after his own children..
It looks like he's been a pretty useless partner and father. Even now, when he's meant to be throwing everything at getting your relationship back on track he's useless.
That is very telling. Right now, you are separated because of his behaviour, and he needs to show he can make changes in order for you to resume your relationship. So you would expect he would be doing everything in his power to reassure you, to show you that he is a good partner and father. That he can be different. Showing you he loves you, doesn't want to live without you.
And yet how is he acting? As though he isn't really that bothered. The phone thing is a good example.... If I'm away from home (for a day or a week end say) and my phone battery is low, I send a quick message to DH or DC to say so. 'Battery's low.. charging now but won't be with it for a bit'. Yet your DP didn't think to do that. For several hours!! He either wants you to feel insecure or he just doesn't think about how you feel.
He knows the damage his previous week end away did, and yet doesn't think about how you might feel this time, if he goes off-radar for a few hours when he promised not to.
His treatement of you is awful. Either a) he thinks he can treat you like rubbish and you will stick around so he doesn't have to bother or b) he isn't bothered whether or not you stick around or c) he is bothered on some level but something else is more important to him (drugs / another person for example).
You deserve so much better. You are not being controlling or a cow. You have had a reasonable response to being lied to, cheated on and gas lighted. His behaviour caused all this and he's showing you he isn't capable of, or willing to, change.