Working on finances as a couple and holding each other accountable doesn't mean you don't trust him even though in this case you have every right not to trust him because he already messed up even before this situation.
To me the fact you can't even come together to discuss your finances like a couple working together says a lot about the communication and relationship and yes you are right to walk away.
But my point still remains even if you get into another relationship unless you want things at arms length, no commingling etc you need to be working together, setting budgets together, making decisions together etc not just say he committed to x and I didn't check for 3 months if not it would be like I didn't trust him. That's not how a partnership works.
I do wish you all the best and I'm glad you're in a good place financially so you're not dependent on him and have the option to make the best decision you think is right for you.
Here is an article about money and relationships, maybe it will explain my point better:
https://www.guardianlife.com/financial-strategies/couples
"Finally, you should commit to having regular discussions about money, lifestyle, and spending. Make it a point to periodically review your overall financial life together, including goals, account details, and asset allocations, to ensure you are making informed financial decisions and staying on the same page as a couple. Circumstances and needs are sure to change — for better or worse — throughout the course of a marriage. But whatever issues you face, regular communication will leave you better prepared to address the changes and handle the challenges together."
So yes he failed you and he needs to own that but I also don't agree with your approach if not having any discussions or checks for 3 whole years under the guise of "it would show lack of trust him". That's not how finances and relationships work, that is burying your head in the sand and pretending everything is fine when deep down you know it isn't.
That is avoiding difficult conversations instead of building a stronger relationship where everything is on the table and you're both on the same page or you realize earlier (preferably before you had a baby) that you're not compatible and walk away instead of waiting until you have a baby and 3 whole years already viewing houses before you realize he hasn't even saved anything. These are the things that form a solid foundation for a relationship and is the difference between a solid relationship where you work together and roommates sleeping with each other and having kids.