BranflakeGirl been there, really feel bad for you going through this, especially while pregnant.
Missus84 has already made some good points. And I totally sympathise with you being nervous - we lived next door to a dysfunctional family falling apart with 2 young dealers who were causing chaos. Reporting it to environmental health and the police resulted in inflaming the situation. These aresholes know that unless they're caught in the act - ie., with drugs on them, or turning up the godawful shite that passes for R'n'B back up the moment the cops are gone - nothing can be done. They made our lives miserable, absolute hell and we moved in the end, from a much loved home that we had put a lot of work into.
However, there's some good news. Your housing association has something that is known as a DUTY OF CARE, to its tenants, and to the community where their housing is. They are meant to ensure that there is no anti-social behaviour. How effective they are depends on how pushy you are prepared to be. I would suggest visiting your local CAB, and seeing if you are eligible for legal aid, and if so,then getting a solicitor who specialises in Housing Law to contact the Housing Association. In the meantime, keep a really detailed record of every disturbance, no matter how trivial it may seem. Also, see if your housing association have other properties they can move you to. You are pregnant - make light of this and insist it is not good for your health. Get your GP to write them a letter too if necessary. Are any other neighbours sick of these people? If so, you may have some strength in numbers - it will also help you feel less nervous.
I am afraid IME, the law is a complete and total arse, but if you can get a solicitor involved it can show how serious you are. While it may do your nut in, under no circumstances resort to their behaviour ever, because believe you me, they'll find a way to use it against you.
HTH, really feel for you, but you have rights and you shouldn't feel uncomfortable in your own home.