They wouldn’t mess up your child, but the baby may be born addicted and need 5 day minimum monitoring for NAS, which not everyone is comfortable with at all. However, I would suggest you phone the GP and also see if you can speak to a different midwife. I know it’s a nightmare advocating for yourself (it gets exhausting doesn’t it) but it’s so important you get support around this. It’s atrocious you haven’t been taken seriously. I’m not sure where you are, so access depends on area, but there are perinatal psychiatrists who’d be able to talk you through options. If you ask your GP they should be able to refer you, they can also refer on to a CMHT for specialist MH support where the psychiatrist prescribes (gp doesn’t always have the knowledge). Alternatively if your gp won’t refer, phone CMHT and explain, they may be able to get you on their books. If you can afford some private counselling sessions, I’d look into those, but otherwise GP theoretically can refer on. Most areas only offer CBT for Depression on the NHS. Some charities also fund therapy, Mind does in some areas, Anxiety UK offer therapy (anxiety often a part of an/pn depression) and may have some good advice as a charity, Rethink offers advocacy which might help you get a midwife to listen.
One thing I would say is that you may be in a better position with anti-depressants and your safety is paramount. Depression around pregnancy tends to be hormone based, but there are some techniques counsellors would support you with that might help - it just takes a long time. As the antidepressants can take months to have an effect, most docs would recommend starting now to be in a safer place once baby is born, however, it is always your choice.
I really hope you get the support you need. Ask your husband to advocate, no matter how lovely he is, he needs to be taking the bull by the horns right now and not allowing no support to be an answer. He needs to phone GP and midwife and voice his concerns until he’s listened to.