Yes we did that, had a baseline size (enough bedrooms for kids to have one each was the crux of it) to keep the mortgage smaller so we could have the lifestyle we wanted.
We outgrew it very quickly though and Covid showed us we were probably too cautious, we ended up upsizing to something more appropriate. That said, we have since had promotions and could go even bigger now if we wanted, but we won't, we feel we have struck the right balance now between house size (4 bed detached for 2 kids) and holidays/financial comfort.
My advice would be to not be too cautious, think about your career trajectories and ensure you have a realistic "baseline" for what you want from your home, and weigh that up with the lifestyle you want.