I am afraid l am going against the grain here op...
I had 2 Dcs, 16 months apart, and don't accept it is acceptable to be hurting your or other Dcs.
My 1st once or twice tried to push the baby away hard and hit out and it was very firmly stopped.
She was taken away , it was explained why that was NOT acceptable and the consequences if it happened again.
Second time was to test said boundaries, so consequence was time out in her crib for 5 minutes on her own.
Problem was solved, she didn't ever hurt anyone again. She was 20 months old but quite capable of understanding.
I don't believe any agressive behavior should be tolerated, get a secure space for Ds to go and calm down, playpen or bed. Use it to put baby down for a time and give yourself a break.
When he hurts other Dcs what is the consequence?
I would take him to one side, tell him what he has done is wrong, make him apologise, if he either didn't apologise or did it again, l would go straight home.
You are by no means a crap mum, it is hard work with 2 small ones but, IMHO boundaries and set consequences makes it all easier.