I came off of sertraline before ttc on advice of my (male) GP. I'd been on it for 8 years, and reduced over 6 months - we had been in Mexico on honeymoon so I was very concerned about timing given the Zika outbreak - and tbh it gave me a dual purpose while waiting before we "officially" started trying, along with getting all my supplements in!
He advised me that as little medication as possible was the recommended approach - and should I conceive and struggle with my mental health there was lots of help and support available from the midwives.
I actually got pregnant very quickly and the first few weeks were a blur of emotions and excitement.
I started feeling properly concerned about my mental health again after 3 months - and mentioned to every midwife, asking how I could get the support my GP told me about. The midwives told me that the only extra support they knew of was referral to a psychiatric specialist unit - to put it bluntly, for people who are effected by mental illness much more severe than my own.
I had good experience and techniques from having had CBT previously so was able to cope relatively well, I had good weeks and awful weeks and put it down to just pregnancy, though sometimes it's been hard to know what's me and what's "normal"... you read about pregnant ladies being hormonal and emotional and sometimes it was like it could be that, and other times ... not so much 
I had a really bad week 31, and went to see my GP again. My regular doc was on holiday - so I got a new (female) GP.
Oh. My. God. Whole new story.
She was fantastic. She listened to me, properly, and explained that the mother's health and wellbeing is just as important as baby's - and if I felt I needed to go back onto any dosage of sertraline, she would absolutely support it as it was one of the safest drugs for use during pregnancy. She didn't say it out loud but I think she was concerned that I was pressured to come off of it in the first place- I wasn't, but had I had her advice earlier I would probably have gone back onto the tablets as soon as I'd started struggling in month 3. I've not enjoyed being pregnant - I have been physically well, but there are times when my mood has been so low, its been pretty shit and I feel like I'm unbearable to be around sometimes.
I feel bad to suggest that the male vs female approach was different- it could have been an age thing, or just their personal view- and I can't believe the difference in knowledge around maternity service and mental health provision in our area! Had I been able to receive that reassurance earlier it would have made a massive difference. Not to baby - but to me.
I'm 34+1 and have decided to continue without - it may be silly, but I feel I've come this far so want to keep going - however I now have more regular midwife appointments now I'm further on, and have moved to have the new GP as my primary contact so I feel much more supported. I've discussed going back into a lower dosage once baby is here - as it's safe during breastfeeding, too. But even now feeling more in control and knowing the option is there is helpful. Especially as I worry about being higher risk for PND and I would like to be a happy, healthy mum for my wee one!
Sorry, that was a long winded way of saying it's totally safe 🙈
I also think it's a very personal decision. If you wanted to try without - it's up to you - but I think if I had regular access to therapy or a group to chat with maybe, which I could have received a normal referral for - it would have made a difference too. I felt really alone, and you don't have to be - just make sure you have a supportive unit around you, it makes all of the difference!
Again sorry for the long reply. I'm at peak insomnia / needing to pee all night time just now and got a bit carried away 😂
Best of luck xx