He's a useless fucking fucker.
You need to honestly reconsider your future with this loser - how dare he both undermine your ability to work while at the same time saying he wants to drop a day at work himself, so he can have time to himself? I mean for FUCKS' sake, how fucking selfish can he be?!
You do need to get away from him, both for your own sanity and for your baby's safety. If his own sister won't allow him to mind her kid(s) then I wouldn't be leaving your baby with him again either. The thing is that the baby will pick up on the aggression, whether or not it's directed straight at him - and it will make him scared, and he'll cry more, which will aggravate the situation more. You run the risk of this fucker either hurting the baby, or going off and leaving him in a dangerous situation - neither of which is going to be good.
Please just sort things out, go back to your parents/mum/whomever and work out how you can get a new job and a new place to live before something awful happens.
And do not leave the baby with this twat. I'm sorry to be so rude, I expect you do still love him but honest to god, I've seldom heard anything as pathetic as his "reasons" why he can't possibly look after his own son.
Ugh. Nothing attractive there.