I would love to say, 'you're overreacting' but yeah, it comes with a small risk, conceiving later. I delivered DC2 on my 38th birthday. My blood results were great. No risk at all.
A year later, at 39, I was pregnant with my third child and the pendulum totally swung the other way. I had like a 1 in 150 or 160 (can't recall now, it's 13 years ago) chance of having a baby with Downs. At the time, the consultant felt I didn't need CVS (the diagnostic test that can be done earlier than amnio, between 11 and 14 weeks) because I was on the cusp of high risk, not actually, firmly in high risk territory, but straddling the line. I was very concerned. The change in numbers between the two pregnancies was shocking. I didn't feel confident at all, going from 1 in 11,000 to 1 in 150 between two pregnancies.
My 20 week scan (done at 24 weeks because of hospital delays) showed all sorts of soft markers which led to a second consultant-led scan a few days later. The consultant performed an amnio on the day which confirmed Downs.
I had my last baby at the age of 42 and, because of my history, had a voluntary CVS somewhere between 11 and 13 weeks. I didn't even do the bloods to see what risk of Downs/trisomies there were. Just went straight to CVS, which was totally normal. He just turned 10 years old this week!
I just want to add that my friend, at 39, had a 1 in 60 risk. That baby had no trisomies and has just finished law school. So, it can be that the risk is very high. But that doesn't mean you'll be dealt that blow. It's 'risk' not 'fact'. In my case, the 1 in 150 risk became a matter of fact and I had to face that sadness. It happens and it's as sad and worrying as you can imagine. But life is risky and difficult... but overwhelmingly, it is hopeful.
So it's hard not to feel that fear because it's an unfortunate reality that once we hit our late 30s, the risk increases. But still, it's very rare and mostly, everything works out great. Hang onto that. Try to keep the worry at bay. That's all you can do.