I’ve done this. I was 7 months pregnant when it started and ds was 3 months when finished. Bedrooms were untouched but we had no running water or toilets downstairs, limited to one habitable room and no kitchen. We coped okay but the dust was exhausting, the noise awful (I invited myself to friends when the hammer drilling happened) and the lack of privacy (couldn’t wander round bf or in my dressing gown comfortably) intrusive.
However, I was there to ‘encourage’ them, we had a good working relationship with the builders due to my availability and it satisfied my control freakery 
We could have delayed the build another 6-9 months (next spot with our builder) but I didn’t fancy a more mobile baby that noticed the noise/disruption
I had lots of anxiety on my first born, and I can say that all the building work, in my experience, distracted me enough to not have that too second time round! So a transfer of worrying to something with an end point (building work)
Big one went to preschool at that point, and the baby distracted us both on the times he wasn’t there.
I wouldn’t recommend it necessarily, but it was better for us to have it done then. The torment is all over and our house is lovely
I would make sure you have the money to finish and if at all possible, the money to Airbnb locally if it gets too much. I seriously considered this a couple of times!