FWIW, my thoughts are -
It might be worth trying to separate the issues that are troubling you as far as you can.
Fertility does decline with age - but generally this is associated with the 35+ age group, with fertility getting lower with age (so at 32, I don't think this is currently a factor for you). It's also an 'overall' picture, i.e. some women may take 2 years to conceive, but others might be lucky enough to fall pregnant the 1st month of trying.
This age related average decline is also something that you might think about if you feel very strongly that you want more than one child.
As several others have pointed out, no one ttc for the first time really knows what's going on with their insides! - most women won't have significant trouble conceiving, but for those that do, it's obviously true that the earlier they start ttc, the more time they have to try and sort out any problems.
It's worth saying out loud that age and fertility is such an emotive subject, getting our heads round what it means for us individually can be really hard. There's a natural tendency for us to believe and pay attention to the stories that suit us, and ignore/get upset and angry about the stuff that doesn't. What we really want is a personalised crystal ball, but what we have are statistics and averages, and very little way of knowing in advance where we will fit in. I say that not to be gloomy! but to say, I understand why you are worrying.
As far as life plans go, and how 'ready' you are to have a baby - personally, I think it's a bit too glib just to say 'there's never a good time.' There are certainly plenty of bad times - a glance down the social services register in any UK city will tell you that much.
I think it's very reasonable to want to be ttc and hopefully pregnant at a time when you can (mostly) really look forward to having a baby, rather than feeling scared and fearful of the changes to come in your life. I think it's true that no one ever feels 100 percent 'ready' and confident - but I also believe that some times really are better than others. That might be because of family relations, emotional/mental health reasons, practical financial circumstances, etc. These will never be perfect! but they all have a big effect on our lives.
When I said it might be an idea to separate the issues troubling you out a bit, I meant that time spent calmly mulling the 'life situation' side of things rather than the 'fertility clock' might be the best thing. Only you know what you want! and what is possible for you. If you aren't ttc right away, then there is time to do a bit of a 'life audit', and look at everything from your support network of family and friends to your work/income situation, and just ask what you feel you would really want to change if you were to have a baby. I'm not saying 'try and plan so everything is perfect!' at all. Just - if you don't feel you are ready right now, then what is stopping you? (and if it is your work/money situation, there are lots of active things you can do to re-organise that).
Sorry for the long post. A shorter reponse would be - 'try not to worry about the things you can't predict/change! but why not spend some time thinking honestly about what you really want to change in your life before you ttc.'
Good luck.