You risk losing career progression, especially as jobs change so rapidly due to technology. The role you were doing a few years ago might look completely different today. Securing a stable office job is much harder now than it used to be.
You also miss out on pension contributions, which is a significant factor. Losing 5–10 years of contributions and compounding can be the difference between retiring at 58 and having to work until or beyond the state pension age.
Generally speaking, supporting a family on a single income while working can be more stressful than being a stay-at-home parent. Even when the job itself is manageable, workplace environments can be toxic, which can create tension and resentment between the working parent and the stay-at-home parent.