So, he walked out on you and his child because - he was jealous of his helpless child getting the attention every baby needs.
He, a grown-assed adult, chose to sulk and leave you to cope alone. He chose to punish you for being a responsible adult who balances the needs of a newborn against the wants of an adult and rightly finds that the needs of the baby heavily outweigh the wants of a petulant self-absorbed adult. Well, 'well done', him!
No wonder you have anxiety!
"I want us to work, I love him, is this common of men? How do you balance it all?"
Serious question - what do you mean by "work"?
Because if by 'work' you mean some rosy notion of happy families where you and he co-parent and raise a healthy happy child; no, it won't work - he will make sure of that.
If by 'work' you mean that you become subservient to this selfish wanker's desire to be the centre of your universe, struggling to raise your marginalised and increasingly unhappy child with the scraps of time he will allow you to have away from serving him - oh, come on! This relationship is doomed, mainly because it's not a relationship. It's a selfish arsehole putting demands on another adult that he does not consider his equal, that he does not respect, that he may not even like (never mind love)!
Which leads me to my next question - why do you love him? What is loveable about him? He left you high and dry - I don't see that as loveable in any way. Has he persuaded you that it's all your fault? I already know the answer it's in his whiny complaint that you "have not put our marriage" - by which he means you have not put HIM - first. He's telling you his leaving is your fault (it wasn't) and that by putting your marriage (him) first everything in the garden will be lovely (it won't). He is such a wanker.
"He has returned and I don't know how I feel about myself and everything."
Which is exactly where he wants you - confused and vulnerable, all the easier to manipulate.
For your sanity, for your future, for your child's future - do not let this man back into your life. He will make you more anxious and unhappy than you are now - all the better to keep you off-balance and subservient.