I had awful anxiety when pg with DS who is now 7. I had had anxiety for years, but it went up a notch while pregnant. Not helped by a high risk pregnancy and the fact that we assumed we could not conceive again. We had 15 year old when I got pg with DS and had given up on the idea of having another.
I too was convinced something would happen to him, even before it became apparent the pregnancy was high risk. I think this was because he was like our 'miracle baby' that we didn't think we would have. It sounds like this may be the case for you too. My baby was (and still is!!) absolutely fine and wonderful.
Against all expectations, I fell pregnant again when DS was 4 and again my anxiety went through the roof. By this time I was actually agoraphobic (still am) and needed someone with me at all appointments. I, like you, was really scared to say anything in case they got SS involved. I was actually convinced I was going to die and said silent goodbyes to my family in my head.
I had to tell them (because my BP would go up!!) and they were fine. They arranged to visit me at home and gave me options for a home birth. No one went near SS. I had my husband then 19 year old daughter for support and I don't know if that made a difference, but they were all lovely and accommodating.
Sorry, that was a bit long winded!! What I'm trying to say is that hormones can increase your anxiety, it is normal to feel something will go wrong and your midwives will not judge you because you are struggling. Congratulations on your pregnancy 