@SnowWhite1985 - I'm with @claireb707 all the way... he IS being horribly manipulative gaslighter - and it sounds like that's been a pattern in recent years.
I know it's always more complicated than you can ever explain on a forum, but from your descriptions I genuinely think you'd be better off without him.
The children issue will always cause resentment and drive a wedge between you, one way or another. Him because he sounds like a big child himself (craving attention, wanting everything his own way, fantasising about a future for himself alone). You because you'll always feel like you're dragging him along, and having to bargain with him for what should be something that fills you both with joy.
Oh, and throwing a glass at a wall when you're alone is NOT a big deal, in my view. His manipulation and passive aggressiveness are far worse. If he mentions it again, I would simply ignore it and change the subject.
Generally: Move back home - do you have a spare room? If so, one of you sleep there initially. Then calmly set about the process of separating your lives. Did you say you own the house? Are you married? Either way, I think he should leave.
Then you can get on with your life. If anyone IRL asks: You broke up because he's a selfish sh*t who doesn't see you as a priority and doesn't picture a future with you.
Good luck and please do pop back to let us know how you're doing. xxx