Oh God, we did this. Bought a two bedroom terrace in London, spent five years renovating it. Finished it in June 2014 and it went on the market in August 2014. We were trying for children from mid 2013 and knew that we'd outgrow the house within a few years.
We then bought a much bigger project - a 3 bed detached with potential to convert it into a 4 bed with a huge kitchen extension. We're now about 2.5 years into the project, it's around 25% complete, we have DD1 and I'm 3 months pregnant with DC2.
Is it worth it? Right now, NO! I'm not sure what your situation is but it's sooo much harder and slower with a child as I can't help and do all the childcare whilst DH does all the renovation work at weekends.
That said, we are moving on now and will be employing much more professional help for the extension and beyond. Also, it really will be the forever house (20 years plus) as it has so much space to grow and this bloody project has almost broken me so I never want to move again. "Deferred gratification" is my mantra.
I think we'll have a few more really difficult years but then I can be a smug bastard. 
. Also, there's no way we could have started this project now (with this level of mortgage) as I'm sure our income wouldn't be high enough (with my part time salary and childcare costs).
I'd probably say through gritted teeth that you should take the plunge (depending on your situation with children and finances). Good luck with whatever you do!