His behaviour kind off sets off alarm bells (to me) for drug addiction. The lying, borrowing money, lengthy times locked in the bathroom, constantly letting you down. The mood swings. Only committing to plans on the actual day, never in advance.
Used to be an addict myself, and I engaged in all of these behaviours. The same behaviour one would display if engaging in an affair.
Either way, he's a selfish arse. You on the otherhand are a full time mother of 3. Dealing with your own difficult mother. Like everybody has said, he's sucking the joy out of your life. An emotional abuser. Have you looked into counselling? I too have a narc/emotionally abusive mother. Kept on choosing men of similar personality.
We gravitate to what's familar, even though it's shit. You can learn how to have more healthy relationships. I found being alone (but I stress, not lonely) whilst going through counselling helped me set my boundaries. Then I met DH, who I love with all my being. Before that I'd had 2 fucking dreadful relationships. Which I gave 15 yrs to, between the both of them. I didn't know I deserved any better. If I'd stayed in those "relationshits", I'd never have met him.
Hear me when I say that you do deserve better. Hoik up ya Big Girl Pants!