Different scenario (we only have 1), but didn't want your post to go unanswered.
My pregnancy was totally unplanned. We were never not sure what happened ... (barring the obvious, of course! ).
I was pretty devastated to begin with. Cried all day when I took the test (3 times - just to be sure...). Couldn't bring myself to tell DH until the next day (he had a thing that evening, and I didn't want to spoil it). We were both quite negative: a lot was, and is, going on in our lives, and having a child was very far from our minds and plans.
Selfishly, I was totally paranoid about my career: things I wanted and needed to achieve before having a baby.
I felt like this throughout my first trimester - couldn't believe I was pregnant, really. Had hysterical giggles at the first scan ("oh look! There really is a baby inside me!") - the doctor must've thought I was a total loon.
But suddently, after that - and probably just seeing the baby, our baby, for the firs time made the difference - I felt better.
My pregnancy was really, really easy. And I realised that my baby, our DD, was my little ally, my little secret friend and companion. She decided it was her time: babies come when they want. From quite a bleak place, I moved into a really positive frame of mind.
Now she's nearly 9mo, and every bit as lovely and 'on my side' as she was when she was inside me. Every day we can't believe how lucky we are.
Our lives are still a mess, and it'll take me longer now to do the things I need and want to do. But that's ok now. I'll get there. And I'll have my DD too!
I'm sorry that you're feeling worried now: that's totally normal, I think - even without additional pressures and concerns. But you'll feel better one day - and then you'll be able to look forward to meeting your new DS or DD! Just think! He or she has decided that it's their time! He or she has chosen you and is waiting to meet you!