Yes, it's worth it. You need a life, and not just for the money. It's been my observation that children of working mothers are every bit as happy and secure as those of SAHMs like myself, if not more so frankly. It's also been my observation that having a second, let alone third, child takes more of your attention and time away from your child than the most full-on career. What's more, you need to set an example for your son, and his future wife, of what real life means -- having both a career and a family.
I see you've changed jobs recently, which makes this harder. But maybe wait a while and then try negotiating with your employer to work a bit more from home or whatever. Because frankly, if women with the power to win £200K jobs don't start to push, women with £20K jobs will never stand a chance. I dropped the ball, frankly, partly to be fair for reasons outside my own control -- infertility, a disabled child, a husband with an even busier career and not enough interest in my career. It was nice for the first few years, but it's grown very very old and very dull. And is a terrible waste, not only of my abilities, but of the fact that, again, if those of us who have bigger jobs don't push back against this endless presenteeism, who the heck will?
FWIW, my observation is that the main person who feels exhausted and inadequate in these early years is the mother. The mother suffers, not the child. If you can hang in there, in a few more years everyone around you will be full of respect for your seniority, and you will likely be glad you held on, and you might even have built up enough power to change the world.