We've done it, but I think I'm the most extreme example I know (in real life). We bought a Victorian wreck 3.5 years ago, needing a huge amount of structural/cosmetic work. Since then, my DH has done most of the work himself and we've had two daughters in that time, now aged 2.5 years and 8 months. (Worth noting that we struggled with infertility for two years and I got pregnant with DD1 a month after we bought the wreck 
). We can't afford to move out and completely underestimated the time and money needed (London build costs!).
We have managed to pay for a double storey extension to be built, but we've acted as the builder and project manager to save money and hired in the trades at each stage. In some respects this element has been harder than doing it ourselves and DH has still had to do a huge amount of work between trades.
@Orangedaisy 
and 
to you. It sounds like we're going through a similar experience. I say to people: "It absolutely hasn't been worth it and if I could go back I would have walked straight past this damn house". To which they reply "Oh, but it'll all be worth it". Urgg! I'm also sick of hearing: "But you can make it your own".
It's absolutely broken DH and me and I'm really hoping we're through the worst of it so our relationship survives as it's really come under pressure. I feel so much responsibility to look after the children 7 days a week (am on maternity leave) and if I ask DH to help then I know that all work stops. We're still 3+ years away from completing the house, have pretty much run out of money, and it feels like life is on hold. On the plus side though, if DH and I can get through this then I think we can get through anything. Also, it's our forever house so I never have to go through this again and will happily die in this house. 
Sorry OP, that's quite a monologue! Absolutely do it if you have the money, can move out and have a very reliable builder or trades to get the job done quickly. Good luck with whatever you decide!