If you can afford it OP then yes do it.
Do you own your own home- could you continue to pay the mortgage on a reduced salary?
If you pay rent, could you continue?
Would you have to return to work and if so, could you afford the childcare.
Do you have a support network?
I only say these things because you posted on AIBU asking and I think these are valid points, however people don't always consider these when starting a family.
Circumstances do change after a baby too, my OH and I lost everything when I was pregnant with DC2, no money, job or house. With the help of benefits we slowly built ourselves back up and are now sufficient and no longer in receipt of them.
I say this because, even though it may never seem to be an ideal time to have a child, you shouldn't put it off if that's what you want. If you were rich and had a partner, you could still lose everything and not be able to provide. It happens. Go with your gut and if you want a child that badly, then do it